Centralising data sources and enhancing integration through the delivery of a cloud-based data management platform

Responsible for maintaining some of the UK’s most used natural water resources and a wide range of heritage buildings across England and Wales, our customer is a highly respected, UK-based charity.

They collaborate with volunteers and communities across England and Wales to create spaces where local people want to spend time.

Pain points

Whilst the client already had access to an extensive and sophisticated data landscape, the information was spread across many disparate sources, which was creating inconsistencies with regards to data extraction and usage.

Data integration had previously been performed through the use of legacy technologies but these methods struggled to meet the evolving requirements that the deployment of modern solutions created. As such, some teams had developed their own separate processes, moving away from a centralised data management approach.

The organisation’s desire to integrate data centrally meant they required a consolidated approach that the entire business would follow. The main challenges associated with the existing strategy included:

  • Data being stored across a number of disparate systems
  • Duplication of data sourcing and transformation with inconsistent approaches and methodologies
  • Multiple independent teams with differing technology requirements
  • Limited data governance and security due to the lack of centralised control
  • Inconsistent use of common data sources resulting in no single version of the truth
  • Inconsistent data lineage and reusability meaning data was shared between departments without a clear strategy
  • Legacy technologies restricted the use of modern data integration methods
  • Reliance on on-premises technology and servers resulting in increased costs for support, backup and DR

Objectives and approach

Circyl was tasked with designing a strategy based around the centralisation of business information within a single platform.

The first phase of the strategy was to create the core elements of the ‘Integration Hub’, along with key integrations that would deliver a usable and cost-effective solution. Given the time-sensitive nature of the project, Circyl worked quickly to develop a solution that would deliver immediate business value.

It was important for the project to provide a greater understanding of what a clear and consistent company-wide data management and integration strategy would offer, and how it could become the basis for future phases to be iteratively built upon.

Through the implementation of an Integration Hub built with modern cloud-based technologies, the client would have access to a flexible data management solution that could adapt and grow over time, allowing it to evolve alongside the business.

Outcomes and principles

The Integration Hub project delivered a solution based on a core set of design patterns and principles that allowed the businesses to understand the benefits of adopting a consistent and centralised approach. These key advantages included:

  • Load once, use many: Required datasets from each source are loaded into a centralised data platform consistently. Centralised data is then consumed by any number of users
  • Data lineage: Managed data contained within the Integration Hub cannot be used directly from source and must be accessed directly from the Hub. This ensures data lineage is fully understood and controlled
  • Source system decoupling (business context): Data managed through the Integration Hub has a business context that decouples data from the source system and removes the direct dependency on data in the Hub matching the source system
  • Support direct database and API sourcing: A modernised technology stack allows users with differing consumption requirements to access centralised data. Whether via direct database connectivity or API requests, teams from across the business have access to a single source
  • Single version of the truth: The Integration Hub now provides a unified and consistent strategy for data supply and the ‘load once use many’ policy ensures data consistency across all the customer’s teams

Technologies utilised

During the project, Circyl leveraged the following technologies:


ASP.Net Web API: An extensible framework for building HTTP based services, including RESTful APIs, that can be accessed in different applications on different platforms such as web, windows and mobile.


Azure Data Factory: ADF is Microsoft’s cloud-based integration tool for managing the extraction of data into Azure from a range of sources. The tool allows configurable pipelines to be developed that automate the data refresh process, keeping report data up to date and removing the need for manual effort.


Azure SQL Database: Azure SQL Database is Microsoft’s cloud-based service that provides the database platform for the centralised data warehouse. With its range of service tiers that can be scaled up or down as demand dictates, Azure SQL Database provides the ideal combination of reliability and flexibility for an evolving data platform.


OData: A well-defined protocol for building, querying, and filtering data via a RESTful API. OData is a best practice way to describe the data and the data model for easy interoperability between data sources, applications, services and clients.


Open API: A standard, language-agnostic interface for RESTful APIs which allows both humans and computers to discover and understand the capabilities of the service without access to source code or documentation.


Power BI: Power BI is Microsoft’s cloud-based enterprise and self-service analytical solution that provides data modelling and visualisation capabilities across a range of data sources. 

Want to know more?

Circyl offer free consultations to allow businesses to discuss their productivity issues and ask us in-depth questions about our services and pricing.

Call 03333 209 969 to speak to a member of our team directly or send a message by filling in the contact form here.