One of the most difficult test activities is performance testing. Nowadays, when applications are more and more complex, it's even harder. Tools designed a few years ago are no longer sufficient.
It all started from a project with unusual customer requirements for measuring performance of a web application. The basic requirement was to examine the responsiveness of the client application from the end-user perspective and the most real values.
We started using JMeter as a tool to measure web application’s performance. Unfortunately, we quickly discovered several problems. As you know, JMeter is not a browser and is used to measure server response time. Comparing JMeter results with the time measured in a browser, the differences are close to a few seconds, it’s too much.
We turned to WebDriver Selenium, which is used for automated testing. It runs on the user interface layer, so it simulates user actions and solves problems that have arisen with JMeter. But there is nothing for free, Selenium WebDriver does not have a mechanism to measure time and generate statistics like average or 90th percentile. The biggest problem is the inability to simulate multiple concurrent users.
Our final solution was based on the simultaneous use of Selenium and JMeter WebDriver under the control of the TeamCity Integration Continues server. TeamCity is responsible for:
- Preparing a package with the latest version of the JMeter script and Selenium WebDriver tests
- Running JMeter tests in the background that simulate load on the server, by sending an appropriate amount of requests
- Running WebDriver Selenium tests parallel to the JMeter that are based on performing actions in your browser to fully replicate end-user behaviour
- When the tests are completed, WebDriver Selenium and JMeter statistics (average, 90th percentile) are collected.
- Next step is to display metrics based on the collected data
- Last but no least is the publication of a trend, where we can see the changes between runs.
The main part to fully automate the process was to develop a module for data collection. You can grab it as part of Selenium WebDriver framework from here. As well as adding extensions to the TeamCity responsible for integration with JMeter, publication of partial results and trends available. You can grab it from here.
To sum up, we did it. There are different ways to solve this problem but we decided to use the well known JMeter and Selenium and it was a good decision.
This subject was presented by Objectivity at the Testing Cup 2015.