Reference Architectures - When Business Intelligence Meets the Cloud

Your company can move ahead of the competition by taking advantage of the power of data—to do so, it’s a good idea to start looking around for technologies that could help you achieve this goal. However, certain technologies work well together, while others not so much. In this article, you’ll find answers to questions regarding how to effectively approach the Business Intelligence (BI) world, and what kind of tools—such as reference architectures—you could use to provide better reporting for your company.

Put Your Data Needs First

The Business Intelligence world is growing rapidly—seeing new concepts, new architectures, new software, cloud innovation, and much more. On top of that, many of these technological advancements are carefully veiled under popular buzzwords, masking their underlying complexity.

When we look at the BI landscape from a technical perspective, we see that many providers try to force companies to change their business processes, so they align with the proposed technology. Will such an approach be able to meet your company’s actual needs? Probably not.

The correct approach would be to first carefully assess your business processes, how they work, and what kind of data and information they have and need. What is the velocity of the data that flows through your company? What volumes of data do you have or would like to have? What is the variety of the data? Another important question that needs to be asked is how do you want to access the information that will be extracted from your company’s data?

You should only proceed with developing an overall architecture for your BI solution once you’ve answered these basic questions and aligned your needs with your business processes. After the architecture is chosen, it’s time to choose the most efficient and cost-effective technology to do the actual heavy lifting.

Reference Architectures—Should You Use Them?

There are architectures that excel at certain types of tasks but are cumbersome if one would like to use them for other types of tasks. It’s the same when it comes to choosing various technological solutions to work together in the hope of rendering our organisation more competitive on the market.

Certain technologies and architectures fit nicely together and others not so much. Many companies create reference architectures that present how one could put pieces together that will work in a given case.

Reference architectures aim to provide guidance—they exist to help you make a decision, but please remember that you don’t need to stick to the reference architecture if it doesn’t suit your business needs. Nevertheless, deviations should be well-thought-out and fulfil your needs better than the blocks proposed in the reference architecture. Reference architectures also facilitate quick project kick-offs and Proof of Concept (PoC) development. You could treat them as guides that allow you to reduce time and costs.

BI Reference Architectures in Azure

When you search for “reference data architecture azure” online, you could become somewhat overwhelmed with the amount of different diagrams you find. You’ll find the most current and recommended one on the provider’s official site—check out Microsoft’s Azure Architecture Center: https://aka.ms/architecture 

Looking at the architecture's browser (available here “Browse Azure Architectures”), it’s easy to become overwhelmed by the multitude of examples. If you’re interested in BI reference architectures, go to the “Databases” section.

When it comes to BI and data solutions, you should start from “Modern Data Warehouse Architecture”. This architecture represents a standard reporting solution—ingest data in batch processing (the refresh rate takes closer to minutes and hours rather than seconds), store the data 1-1 to the source systems in the data lake, and then process and load into the data warehouse model. Finally, this will enable you to analyse the data through analytical dashboards, reports, or advanced analytics available for all users.

Figure 1. Modern Data Warehouse Architecture, Microsoft

Another page worth visiting is “Azure data platform end-to-end” which builds on the previously described architecture. Modern Data Platform Architecture extends Modern Data Warehouse Architecture by adding an option for the processing of data from IoT devices, sensors, and other stream sources. It also presents a place for the extension of Cognitive and Machine Learning (ML) Services.

This architecture follows the Lambda Architecture principles. It enables both types of processing— batch processing (Data Factory, Azure Data Lake, Synapse Analytics), but also real-time processing (Event Hubs and Stream Analytics) capable of delivering data for analytics in the lowest latency possible.

Figure 2. Modern Data Platform Reference Architecture, Microsoft

A complete description of all components can be found on relevant documentation pages.

Well-described components will enable you to meet your business requirements quickly and with the use of the most fit-for-purpose services. Just a few years ago, defining Big Data architecture in the cloud required long hours of research, analysis, and conducting many PoCs—whereas today, all cloud providers make general roadmaps available.

As long as you’re not forced to be cloud agnostic for reasons beyond your control (e.g. multi-cloud project), adopting the right kind of reference architecture comes down to a good understanding of business needs. Before you make your choice, you should know what’s most important to you—whether that be real-time data processing characterised by mere milliseconds of latency, or something else—either way, reference architectures should meet your needs.

Conclusion

As you can see, reference architectures should not force you to change your business processes. Instead, they should be seen as guidelines regarding which types of technological solutions should be used together to achieve your BI goals in the most time and cost-efficient manner possible.

Reference architectures can help you discover which technological solutions work well together and which don’t. When needed, there are manners in which you can adjust the reference architecture to better suit your needs, but if it happens to be aligned with your goals, it could already be effective the way it is. All in all, reference architectures are a great starting point once your data journey leads you to the cloud.

If you’d like to find out more about how data can help you transform your organisation, download Objectivity’s latest complimentary eBook: “How to Build a Data-Driven Organisation”.