Centralising data sources and enhancing integration through the delivery of a cloud-based data management platform

Responsible for maintaining some of the UK’s most used natural water resources and a wide range of heritage buildings across England and Wales, our customer is a highly respected, UK-based charity.

They collaborate with volunteers and communities across England and Wales to create spaces where local people want to spend time.

Pain points

Whilst the client already had access to an extensive and sophisticated data landscape, the information was spread across many disparate sources, which was creating inconsistencies with regards to data extraction and usage.

Data integration had previously been performed through the use of legacy technologies but these methods struggled to meet the evolving requirements that the deployment of modern solutions created. As such, some teams had developed their own separate processes, moving away from a centralised data management approach.

The organisation’s desire to integrate data centrally meant they required a consolidated approach that the entire business would follow. The main challenges associated with the existing strategy included:

• Data being stored across a number of disparate systems
• Duplication of data sourcing and transformation with inconsistent approaches and methodologies
• Multiple independent teams with differing technology requirements
• Limited data governance and security due to the lack of centralised control
• Inconsistent use of common data sources resulting in no single version of the truth
• Inconsistent data lineage and reusability meaning data was shared between departments without a clear strategy
• Legacy technologies restricted the use of modern data integration methods
• Reliance on on-premises technology and servers resulting in increased costs for support, backup and DR

Objectives and approach

Circyl was tasked with designing a strategy based around the centralisation of business information within a single platform.

The first phase of the strategy was to create the core elements of the ‘Integration Hub’, along with key integrations that would deliver a usable and cost-effective solution. Given the time-sensitive nature of the project, Circyl worked quickly to develop a solution that would deliver immediate business value.

It was important for the project to provide a greater understanding of what a clear and consistent company-wide data management and integration strategy would offer, and how it could become the basis for future phases to be iteratively built upon.

Through the implementation of an Integration Hub built with modern cloud-based technologies, the client would have access to a flexible data management solution that could adapt and grow over time, allowing it to evolve alongside the business.

Outcomes and principles

The Integration Hub project delivered a solution based on a core set of design patterns and principles that allowed the businesses to understand the benefits of adopting a consistent and centralised approach. These key advantages included:

• Load once, use many: Required datasets from each source are loaded into a centralised data platform consistently. Centralised data is then consumed by any number of users
• Data lineage: Managed data contained within the Integration Hub cannot be used directly from source and must be accessed directly from the Hub. This ensures data lineage is fully understood and controlled
• Source system decoupling (business context): Data managed through the Integration Hub has a business context that decouples data from the source system and removes the direct dependency on data in the Hub matching the source system
• Support direct database and API sourcing: A modernised technology stack allows users with differing consumption requirements to access centralised data. Whether via direct database connectivity or API requests, teams from across the business have access to a single source
• Single version of the truth: The Integration Hub now provides a unified and consistent strategy for data supply and the ‘load once use many’ policy ensures data consistency across all the customer’s teams