Skip to content
Pattern - Header - Left Pattern - Header - Left

Improving data management with a Common Data Layer

A leader in professional services optimises the way they process sensitive data with innovative technology

Pattern - Header - Left Pattern - Header - Left
Case Study Improving Data Management With A Common Data Navy 800X400

The Client is an international professional services network with headquarters in the United Kingdom. The company operates in more than a hundred countries and is one of the world’s largest professional services networks.

The challenge

Business Need

The Client is a vast professional services network that processes extensive amounts of employee-related data. They utilise a dedicated HR system and employ a specialised team tasked with creating employee master records. The process of producing such “gold records” was complicated and time-consuming. Moreover, due to the size of the Client’s organisation, there were many analytical teams who created different data marts that leveraged the employee data. Each team had to retrieve and process the data separately, which was putting a strain on the source systems and causing discrepancies. Finally, the Client wanted to introduce the GDPR in their processes to make sure they have proper control of the data and can guarantee its safety.  

The company decided to create a Common Data Layer (CDL) to ensure the coherence and security of their data across all teams. Furthermore, this idea opened up additional possibilities for switching from certain SQL server solutions to a data lake approach and further optimising the cost of their data management. Objectivity, as the Client’s trusted technology partner, was asked to support them in building a new solution in line with their needs and priorities.

The solution

Project Details

At the beginning of the project, Objectivity analysed the Client’s requirements and the vision they had already created for their new solution. Further on, the Objectivity team took the responsibility for the end-to-end delivery of the new system, from design, through the application of a Data Vault model and orchestration, to implementation and testing.

As the project progressed, the Client’s initial vision and plan were regularly amended to reflect all the new steps worked out together with Objectivity. The project team made sure that the entire process was transparent, and the final solution was consistent on all levels. The selected data model allows for efficient adaptation to new and changing data sources, while still being able to process historical data.

Objectivity leveraged the computing power of Databricks and the flexibility of a data lake and Azure Data Factory to ensure consistency and smooth orchestration. What’s more, Databricks provide great versatility in notebook creation as well as a highly scalable and testable Python code. Objectivity was thus able to build both unit and integration tests for the new solution. In cases where an issue was found, the tests were appropriately expanded, so that the production environment could be constantly and thoroughly monitored.

When it comes to the data lake, all data is stored in the Parquet format, which is optimised for performance. In addition, the Objectivity team introduced a comprehensive distribution model that allowed the data to be redistributed from the data lake back to an SQL database when needed.

The Data Vault model was created in a way that specifically addresses data privacy concerns and utilises appropriate encryption. Dedicated satellites within the Data Vault separate the highly-confidential data from other internal data types.

The results

Business Benefits

The CDL solution built by Objectivity was designed to simplify the data management process for the Client’s analytics teams. It provides them with easier access to information through one data source and removes the need to look up and clean the data they would need.

During the implementation, Objectivity applied DataOps, a mature, enterprise-level approach towards data integration management. With its code versioning, unit and integration testing, and deployment automation, the DataOps approach helped to bulletproof the production environment and render it error-resistant.

The initial tests of the solution showed that data processing in Databricks is 30% faster than in the previously used SQL server. Moreover, this solution will help the Client reduce their infrastructure costs.

When all of the Client’s teams switch to using the new Common Data Layer, the duplication of data will be significantly limited, and the company will notice savings in both time and expenses.

You may also be interested in

01 / 03

Contact

Start your project with Objectivity

CTA Pattern - Contact - Middle

We use necessary cookies for the functionality of our website, as well as optional cookies for analytic, performance and/or marketing purposes. Collecting and reporting information via optional cookies helps us improve our website and reach out to you with information regarding our organisaton or offer. To read more or decline the use of some cookies please see our Cookie Settings.