A testing strategy to support legacy system modernisation

Our client, Edrington, the prestigious privately-owned international spirits company, based in Scotland, engaged Objectivity to undertake a transformation of their Bulk Stock Management system based on the IBM I-series platform. We relished the challenge and started a complex yet rewarding multi-year engagement building on a strategic partnership already in place.

Edrington’s Bulk Stock Management system (TESS) tracks spirit from the point of production to the point when it is ready for bottling. This system has been running since 1986 and comprised initially of over a half-million lines of IBM RPG III code. Managing over £400 million worth of stock, the data processed by this legacy system is business-critical and is used to provide key metrics for internal stakeholders and external compliance institutions.

The unique nature of the business coupled with more than twenty-five years of bespoke customisation had contributed to previous modernisation attempts being unsuccessful. The system became increasingly difficult to maintain and support, and with deferred change requests, there was an increased risk to Edrington’s business.

With an incremental approach to delivery, testing was a critical part of the modernisation programme to ensure that the Bulk Stock Management system continued to work correctly.

The challenge

Business Need

With the requirement that the Bulk Stock Management system remained live and in use throughout the programme of modernisation, the combination of both manual and automated testing was crucial with incremental changes having to be ‘right the first time’.

The scope of the planned changes, with the modernisation of every programme and data file, determined that changes had to be delivered incrementally to reduce the risk associated with large code deliveries. There should be no “big bang” implementation.

With limited system documentation, one of the significant challenges at the outset was to create a comprehensive testing strategy that would encompass the various included business processes and the adoption of the existing system.

Adding yet further to this dilemma was that the business processes varied by operational location and that the system was an amalgamation of four different underlying systems.

If the challenges above were not sufficient, there also was a fixed time for delivery with the imminent retirement of the subject matter experts who had maintained the system for many years.

A standard testing approach would not work. We had to focus our attention on the incremental and prioritised modernisation of programmes and their related data files.

The complexity of the legacy design and implementation can cause many challenges during a digital transformation. In this case, the code had been developed incrementally over 25 years, mostly by adding new functionalities, and without any significant code refactoring or technical debt reduction. The existing relationships between system entities illustrate such complexity:

The complexity of the system meant that manual analysis was not feasible in the first instance. Therefore, automated system analysis tools were utilised to identify the relevant entities, their interrelationships and usage. The use of an automated system analysis tool was essential in determining our testing scope and approach.

Process by process, we had to determine what programme was running under the screens or batch programmes. Based on this, we were able to create scenarios to cover all possible paths, including edge cases. These formed the basis of a regression suite of tests.

In determining these tests, and the subsequent scripting of these items, we were able to minimise the risk and impact of the subject matter expert retirement, and to enhance the overall ongoing system maintenance and supportability.

The solution

Project Details

With the lack of existing system documentation, the engagement and the close cooperation of business SMEs were essential. Not only did these personnel understand the business processes, but they also knew the nuances of the system usage, much of which had evolved. We established regular and close cooperation with these business Product Owners. Ensuring that business operations continue while implementing changes was a crucial point in the discussions. These discussions and the definition of test scripts had the added benefit of providing materials for knowledge capture and transfer.

The review and selection of an automated testing tool was central to our testing strategy. We focused our attention on the most important requirements that this tool must meet:

• designed to be used by quality assurance (QA) teams, Business Analysts, business users and developers to support fast functional and regression testing of the user interface (UI);
• the ability to build automation scripts rapidly with minimal maintenance meaning that it was ideal for agile projects, waterfall development and also package upgrades;
• configured to recognise the applications automatically, getting the full content of each screen as it appears without the need to interrogate any individual items, to click on it, or write any programme code;
• ensure that the UI is still functioning correctly and to test downstream from the changed RPG programme(s);
• to automatically log any differences compared with reference implementation.

Our underlying rationale was “to change everything, but to change nothing”. That is, every line of programme code should be modernised, and every data file updated. However, from a user’s perspective, there should be no visible change to the user interface. Therefore, capturing the existing process flows and the system interactions was central to the test automation. We recorded all business processes using the test automation tool. Once changes were made, and by running the automated regression tests, we could be sure of the expected results. Our approach was to validate a detailed business process and then combine the detailed business process tests to verify an end-to-end (Epic) business process. Step-by-step, we confirmed that all business rules were working correctly, and when any deviation was identified, the defects were logged.

Automation was crucial to enable repeatable and timely end-to-end testing. The final stage of testing was user acceptance testing, and with engagement throughout the programme, this was undertaken enthusiastically by business stakeholders. The rigorous and detailed approach executed successfully was well-received by the business and set the expectation and approach for future testing cycles.

The automation of the end-to-end testing of business processes was composed of a series of constituent individual test cases. These test cases were managed on a just-in-time basis to ensure that in every iteration, the correct test cases could be run automatically to provide instant results and fast verification of quality. Phase 0 of the programme delivered sufficient test cases to start the iterative modernisation of the system. This ‘spine’ of test cases was used and re-used throughout the programme. Additionally, related to ongoing support and maintenance, the spine forms part of an overall repository of test cases. 

We engaged business personnel who work with the application daily to conduct acceptance tests. We built a list of the most important functionalities (the test spine) to be sure that each time when we implemented changes, the production environment would continue to function. The use of the test spine and automated regression gave us confidence that the product quality was very high.

User Acceptance Testing (UAT) was an essential part of the incremental release process with sign-off by Product Owners that the modified application worked as expected. Testing executed against test designs prepared by SMEs and testers and validated against the expected results.

Before the first UAT cycle, various workshops and individual sessions with business stakeholders were conducted by the Test Manager to explain, discuss, and agree on:

• UAT meaning
• role of business users in the UAT phase
• responsibilities of UAT users
• test execution process
• test results sign-off process

This approach allowed us to conduct the first UAT cycle very smoothly with full engagement and commitment from business stakeholders.

Following each of the incremental releases, and with their detailed application understanding, the QA team played an essential part in supporting the end users with post-go-live hyper-care. There was no system outage as a result of the incremental deliveries.

Test execution was completed by Business Users and Subject Matter Experts (SMEs) and supported by members of the Objectivity test team. The test team’s role during UAT was to:

• provide testing expertise;
• assist with data setup where required;
• advise on what information needs to be captured for defect triage;
• assist with test execution (if required), at the request of the Delivery Manager;
• be on-site across several operational locations to support business users (Glasgow and Distillery);
• conduct daily stand-up meeting with UAT users to discuss test status, progress, and problems;
• prepare UAT Progress and Status reports;
• gather UAT sign-off from Product Owners.

The results

Business Benefits

1. The Bulk Stock System and several associated applications are now on strategic, supportable, and maintainable technologies.
2. The test spine and scripts provide a viable foundation for continuous strategic improvement and future transformation.
3. Positive engagement and relationship building across both the business and IT disciplines has strengthened our strategic partnership.
4. Cross-team collaboration has been essential in allowing us to deliver successfully.
5. We have addressed the two key business risks of subject matter expert retirement, and overall platform supportability and maintainability.
6. Extended knowledge transfer is undertaken to a new joint (Edrington and Objectivity) product development team.
7. Ongoing platform change enabled, including:
   7.1. New automation test regression suite (114 scenarios, thousands of screens, impractical to repeatedly run manually – it would take weeks as opposed to hours).
   7.2. New Configuration Management process and toolset for concurrent distributed development.
8. Permitted the error-free migration of the Bulk Stock Management system to a new hardware platform.
9. Despite challenging timelines and technical complexity, the team successfully delivered a critical, operational, and modernised application and supporting applications.