How to deliver a successful digital transformation

Supercharging your legacy system – giving it a new lease of life

Deciding to start a digital transformation is tough. The more complex the IT system, and the closer it is to the core of a company’s business operations, the tougher it is! Key decision makers are often uncertain how to approach the challenge. Legacy systems have supported and grown with businesses over many years, but also older technology can create issues with scalability, supportability and reliability. If we add to that a growing shortage of skills in many legacy technologies due to both retirement and lack of new technical resource to replace, it is no surprise many organisations are cautious about taking that step towards digital transformation.

Under the hood, the anatomy of the legacy system

Our Client, a prestigious privately-owned international spirits company, Edrington, based in Scotland, engaged Objectivity to undertake a transformation of their Bulk Stock Management system based on the IBM I-series platform. We relished the challenge and started a complex yet rewarding multi-year engagement building on a strategic partnership already in place. The Client’s Bulk Stock Management system tracks spirit from the point of production to the point when it is ready for bottling. This system had been running since 1986 and comprised initially of over a half-million lines of IBM RPG III code. Managing over £400 million worth of stock, the data processed by this legacy system is business-critical, and is used to provide key metrics for internal stakeholders and external compliance institutions.

The unique nature of the business, coupled with more than twenty-five years of bespoke customisation had contributed to previous modernisation attempts being unsuccessful. The system became increasingly difficult to maintain and support, and with deferred change requests, there was an increased risk to the Client’s business.

Modernisation Principles

Working in partnership with the client to address critical constraints, we defined a set of principles to determine the best modernisation path:

• with business process knowledge scattered throughout the organisation, knowledge acquisition and transfer had to go hand-in-hand with the modernisation;
• there were fixed timelines with the imminent retirement of the subject-matter experts;
• changes had to be delivered incrementally to reduce the risk associated with large code deliveries -there should be no “big bang” implementation; and
• extensive testing was crucial as the transformed system had to be ‘right first time’ using a combination of both manual and automated testing.

Getting race-ready

In creating an impetus for change, several factors were pivotal to the success
of this digital transformation.

Technical complexity

The complexity of legacy design and implementation can cause many challenges during a digital transformation. In this case, the code had been developed incrementally over 30 years, mostly by adding new functionality, and without any significant code refactoring or technical debt reduction. Such an approach caused maintainability and supportability issues as well as making enhancements overly complicated. The existing relationships between system entities illustrate such complexity:

Business complexity

Coupled with technology complexity comes business complexity. We adopted a smart approach to analysis with over one hundred prioritised business processes being potentially impacted. Alongside the business stakeholders we assessed the criticality of each process - the closer the process was to the main flow of delivering the final product, and the more frequently it was used, the higher was the priority assigned. Likewise, prioritisation occurred for regulatory and financial compliance processes. The consolidated view on the processes gave a clear perspective on processes
which required critical attention during the digital transformation.

Business buy-in

Business buy-in is critical for the success of any significant transformation. The involvement of senior business stakeholders and experienced end-users across the business from the outset was essential. To build stakeholder confidence, we ensured everyone understood the project goals and approach, as well as the involvement required from each of them in particular active engagement for both Knowledge Transfer and User Acceptance Testing.

Tuning approach

Throughout the digital transformation, our overriding consideration was that there should be no interruption to business as usual – this system underpinned critical business processes, and the cost and disruption of any failures would be severe.

Both business and technology factors determined the approach:

• individually rewriting the application functions was not feasible as having grown over many years the existing architecture was complicated and intertwined;
• the risks were too high for a ‘big bang’ approach, and the impact of any failure could be catastrophic for the business; and
• the digital transformation provided the opportunity to consider all business endorsed project goals while managing costs and limiting the duration of the project.

The transformation was planned in two phases to minimise the risk to the business:

In Phase 1, the transformation took place ‘under the covers’ with the user interface (UI) being left untouched and with no visible changes to the end-user. With the addition of a temporary (interim) middleware layer, the transformation of the database structure, the system code, and reporting tools progressed. Alongside the implementation of an automated regression test suite, and before the production release, users were quickly able to identify any issues by running their usual daily activities in a test environment. In parallel to those actions, technical and business knowledge was collected into a newly established knowledge base.

Applying the best Agile principles, and through a series of incremental and iterative live deployments, the system was refactored and released – there was no “big-bang”.

After completing Phase 1, and with renewed business confidence and buy-in, Phase 2 has started. With the removal of a change freeze required for modernization purposes, functional changes can now progress, in parallel with code refactoring and UI enhancement to a modern style. As a result of those changes, extensions of the knowledge base and automated tests are being implemented daily. The Phase 1 transformation provides a solid foundation for Phase 2.

Undertaking a major service while still driving around the circuit

The approach

We like to think that the digital transformation approach adopted is analogous to performing a major automotive service, replacing many critical components, but all while still driving around a racing circuit.

We commenced by identifying any data tables which were no longer in use and excluded them from the transformation scope. For the remaining tables, each was assigned to one of a series of planned deployment iterations, deliberately starting with the simplest. Planning with deployment iterations enabled us to continuously assess and refine our approach before tackling more complex system areas. For each iteration, we cleansed the data in each table, archiving the original raw data. The cleansed data then was stored into a relational database which acted as the foundation for
the subsequent code transformation.

Temporary Middleware

Changing a database design usually requires code changes. In this case, as the legacy data and systems were so heavily intertwined, we introduced a temporary middleware layer which acted as a bridge between the new data design and the existing cod. By using RPG’s open access handler to access the transformed database, we were able to minimise changes to the program code while the code transformation was taking place.

Data Cleanse

Undertaken in collaboration with the client SMEs, this task was critical to undertake prior to their retirement to ensure the population of the new tables with the relevant data.

Code

We gradually upgraded and rewrote the program code leveraging appropriate tooling to accelerate conversion to use the new database design and subsequently removing the interim middleware with the writing of replacement program code to directly access the new database structure.

Testing

The testing of end-to-end processes was automated, and dedicated test scenarios for each program created. Automation was crucial to enable repeatable and timely end-to-end testing. The final stage of testing was user acceptance testing, which was undertaken enthusiastically by business stakeholders. The rigorous and detailed approach executed successfully was well received by the business and set the expectation and approach for future testing cycles.

Deployment

The deployment strategy was defined to ensure the highest levels of business continuity, with planning for all eventualities and roll back plans if required. There were no unplanned business outages attributed to this digital transformation.

Knowledge Transfer

With the imminent retirement of the critical subject matter experts, it was crucial to secure business continuity and to acquire adequate knowledge and understanding to maintain and enhance the system. In parallel to the digital transformation, a knowledge transfer programme, designed to capture both explicit and tacit knowledge, was instigated. Existing documentation and held a series of workshops, meetings and interviews all contributed to a wiki styled knowledge base. This repository has formed a cornerstone for knowledge management and continues in use to the current day – it includes application process flows, business rules, troubleshooting articles, interface information and business-specific case scenarios for future development.

Outcomes

The business outcome achieved is a consolidated and modernised platform, fit for purpose and fit for the future. It delivered:

• improved supportability, maintainability and reliability of the system;
• a foundation to provide additional features on the new platform – both to extend and enhance business
  processes; and
• positive engagement and relationships between the Business and IT to assure future operations

In effect, we disassembled the car into its component parts, and these were re-engineered, tooled, serviced, tested, then reassembled, and all this while the vehicle was still moving!

Pit-crew lessons for legacy transformation

2 years into the ongoing journey, we have captured the following insights:

Early performance testing with robust proof of concept is essential.
With an unacceptable underload performance, our middleware layer component did not perform as well as expected for some key tables within scope. From this, we learned that all vital system components must be thoroughly tested (simple and complex use cases) and should be a subject to a proof of concept. Processes with the highest volumes should be validated with performance testing to avoid potential operational issues.

Despite extensive testing, even small data errors or code defects can result in a significant business impact.
Data accuracy and quality is vital as even a minor mistake in a report may result in an incorrect business critical decision. Test design must be comprehensive to validate every feasible outcome (including edge cases). Robust test checking, test automation and a lean communication process need to be in place for the prompt issue detection and resolution.

Relying on analysis only from the User Interface has associated risk.
Where there is a limited systems documentation and majority of the knowledge rests with end-users, their understanding of the underlying end-to-end processes and scenarios may be incomplete. The aspects of the system which are visible to end-users might be a good place to start but additional analysis is required to cover back-end and hidden components which are invisible to end-users. Those include complex calculations, batch jobs, and interfaces.

Edge-cases and variant identification is essential.
Sometimes business stakeholders identified that some processes were used differently in different locations. Elaboration of these cases is crucial as the process differences may be significant, impacting both requirements and solution design. Prompt assessment should occur for any variants discovered during data cleansing and analysis.

The core IT system is a part of a wider eco-system of applications.
Do not forget the readiness of supporting applications. Proper planning, including assessment of dependencies, mitigates risk and the timely availability of all solution constituents. Overlooking a single component might mean that a deployment of a core feature cannot go ahead. Think of the application eco-system as a whole,
and carefully analyse the importance of supporting applications on the digital transformation project. Business analysis is an integral part of the entire transformation process. It might be tempting to assume that transforming a system or application like-for-like does not require the engagement of a business analyst. This ‘saving’ is a false economy; establishing a standard project team with a business analyst, technical architect, project manager, developers and testers is invaluable as each brings specialist knowledge, experience and understanding.

Ways of working.
Many organisations struggle with digital transformation, and some are even afraid of considering it. Despite being crucial and unavoidable to maintain a business advantage, the term digital transformation has become passé with sometimes negative connotations. Objectivity provided our client with the confidence, that whatever obstacles we are going to face, we face them together in the journey. We would even advance the hypothesis that the cultural fit between the transforming organisation and the IT supplier is an absolute must for the transformation process to succeed.

The checkered flag – getting it over the line

Despite potential challenges, there is no avoiding the need for the digital transformation of legacy systems, especially when the system is at the heart of business operations. This need is especially true in regulated industries, where there may be compliance or legal changes. The priority is to understand the underlying complexity from a business and technical perspective and to define a tailored and workable transformation approach. In this programme, one of the critical factors in the programme’s success was our joint effort, along with a shared understanding and commitment.
While Objectivity provided the technical capability, project methodology and domain knowledge from previous projects, Edrington was closely engaged throughout to provide direction, advice and guidance. Problems were shared openly and immediately, and we worked together to find pragmatic solutions. This included leverage of wider parties and organisations to ensure best practice was applied e.g. IBM and a 3rd Party Testing software provider. Finally, we were able to ask the right questions, listen and agree on appropriate solutions whenever we hit a bump in the road. Together with fantastic business engagement, a goal-oriented attitude, and authentic passion from the client, Objectivity has helped make their digital transformation a reality.

About the authors:

Grzegorz Łoniewski
Programme Lead Business Analyst with over 15 years of experience in software development. Greg led the discovery and preparation phase in the project.

Maciej Wawruszczak
IT business analyst and a firm believer in putting business problems first, and secondly selecting the best set of tools to solve it. He was a Business Analyst in a delivery phase of the digital transformation of the existing core IBM-I system for Edrington.