In the below story, I have described a case study based on my experiences with updating JMeter to a newer version. A story full of plot twists, dangerous action and hopefully satisfying climax. And maybe some lesson how to and how not to do update JMeter in Your CI Pipeline.
Once upon a time...
I had a lazy day when with some extra velocity and the boss of the testing guild asked me:
"Listen, we have our Objectivity Templates for JMeter, and they are a little bit out of date. Could you update it to a newer version?" - Tomasz the Grey
Well, I thought it seemed simple, how hard could it be? Seeing no reason against it I agreed. There is one thing that you, my dear reader, need to know: My experience with JMeter at that point in time was just a few hours of playtime, nothing more. So, if you picture a lousy case Dunning-Kruger effectyou are right.
After asking some questions and looking through our Git, OneNote, SharePoint and extracting knowledge from a few people with know-how, I have learned that:
- The last Version used in our templates was JMeter 2.13.
- The newest version at the time was JMeter is 4.0 but apparently, that version was quite buggy. So, we decided to update it to 3.3 with 4.0 being a stretch goal.
How do you update JMeter templates?
Unfortunately, Google is not helpful in this case. When I was trying to find some info on this subject, I couldn't find anything useful, mostly info about templates themselves and how to use them.
It turns out that in most cases, it is as simple as opening the template in a new JMeter version, and it will auto-update it to the latest version.
Of course, it would also be worthwhile updating the template.xml file.
You will have a proper description of the templates displayed in the new JMeter. As for me, the easiest way was to manually copy the content of tags <template>...</tempalte> in the old file and paste them in the new one.
The last thing I had to do was to make sure all plugins were updated.
The easiest way to do it was to open all the templates in JMeter and go to plugin manager. And check for updates.
So, Done and dusted. Time to go home! Right?
Unfortunately, no. There was one part of this task that I hadn’t realised in the first place. The one that appeared to be much more complicated. The issue which consumed a lot of my time to be finally figured out:
I had to prove that those templates were working and to do so, I had to update our CI pipeline to work with new JMeter.
In short, Testing Guild among other things provides an example of Pipeline for performance testing. Each tribe or project is free to use it as a template or inspiration for their performance test.
Of course, not all projects use this as their template, after all, there is no one size that fits all. But still, it is good to have a working example as a starting point.
And Here is where the fun starts.
Objectivity internally uses TeamCity as Build & Integration Server.
We have a bunch of PowerShell scripts to configure our pipeline.
There is another batch for running JMeter.
To make it all work is a challenge, an interesting one, but still huge. So how I've decided to tackle it?
- Copy old TeamCity pipeline and base for our changes.
- Find a test agent to be used for testing. In a way, that won't affect other projects.
- Make proper changes in pipeline setup to support new JMeter.
- Fix all errors that will happen.
- Propagate changes to the rest of agents.
- Document process so it will be easier next time.
It turned out that the Test Architect started doing something with this a few months ago - he never had time to finish it. But thanks to him, the new project has already a setup in TeamCity as a copy of the old one. The next step was getting Agent to use it as a test field. Fortunately, each new JMeter instance won't affect old one so the risk was minimal for other project making it quite easy to find a suitable agent.
So let's move to the next phase
Updating the process
The `build` has 5 stages
- kill all hanging processes - No changes required
- Warm up - changes required (This step runs a few simple JMeter tests to "warm up the environment")
- Actual Tests Run - changes required
- Generate Trends - changes required
- count trends - No changes required
steps 2 and 3 use scripts of objectivity creation JMeter-start
and step 4 uses another of our own scriptsJMeter-Aggregate-Report
Before we start making changes, we will run an original JMeter job to see if it even works. Two tests have failed. I don't like leaving stuff unfixed, but for now, I've decided to leave them. We will create a task to fix it later.
First: We need to change all JMeters steps to point to our new version which for now is on d:/apache-JMeter3.3 - and only on our test agent.
From now on, we will need to specify our agent to run tests.
Just to be safe, I will use a TDD-like approach - 1 change and then run tests.
So after changing JMeter Build failed.
After looking into errors, it turned out, the problem is a java version.
It turned out 3.3 need Java in Version 1.8 we were using Version 7. Fortunately, one was already installed on the Agent and I only needed to point it to the new version.
(Even if that wouldn't be the case you can download java from hereand there is no problem with having a few different versions installed on the system).
Another run and warmup step is failing.
It turns out, the embedded resources can be problematic at times I am reluctant to change properties, but apparently, the only way to deal with it is in JMeter.properites.
# If embedded resources download fails due to missing resources or other reasons if this property is true
# Parent sample will not be marked as failed
Another run another failure:
Exception in thread "main" java.lang.IllegalArgumentException: Tool class Reporter not found
Between v2.13 a 3.3, there were some changes to Cmdrunner - basically a tool that was used for running some JMeter functionality from the command line. In short, all this stuff originally lived in <JMeter-root>\lib\ext now cmd runner is in <JMeter-root>\lib
The recommended way or using cmd runner is by bat or ssh file in bin folder. Unfortunately for me, our PowerShell is calling it directly.
So for now, I've made a copy of cmdrunner (I had also renamed it from cmdrunner-0.20 to cmdrunner) and put it also in ext folder. In future, we will need to update PowerShell scripts.
Another run another failure:
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: kg.apc.JMeter.vizualizers.AggregateReportGui
Do you remember that haphazard plugin update I did? It turns out, the report plugins weren't part of any JMX I used, so they weren't updated.
For this one, I had to update jpgc-synthesis plugin
I will cut short here: There wasn't only plugin missing for me. So here is the list of errors and connected plugins:
ERROR: java.lang.RuntimeException: java.lang.ClassNotFoundException: kg.apc.JMeter.vizualizers.ResponseTimesOverTimeGui
add jpgc-standard set
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: kg.apc.JMeter.vizualizers.BytesThroughputOverTimeGui
add "5 ADDITONAL GRAPHS" - yes it is realy its name
Exception in thread "main" java.lang.RuntimeException: java.lang.ClassNotFoundException: kg.apc.JMeter.vizualizers.ResponseTimesDistributionGui
Distribution/Percentile Graphs add
But there was one more problem that occurred between all those missing plugins.
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
If you are using JMeter 3.2 (or above), you probably got used to seeing this error in the command line and learned to ignore it.
If you look at this error in connection to JMeter, it is mostly ignored by the whole community. As it is harmless.
Well, it isn't in case of our CI. TeamCity was reacting to error sent to standard error output and in the reaction, it was stopping the run.
After doing some reading on the Log4j2 I've come up with a solution, and it was ugly...
What is going on here?
Well, JMeter is working on java.
So I can "inject" the path for the config file directly to java execution.
The problem is, in our pipeline configuration I have no option to provide such parameters.
As a workaround, I can provide a path for java. So, I can do a wrapper for java which will provide this parameter:
I've modified cmdrunner bat to look like this:
Java -Dlog4j.configurationFile="<configfilepath>\log4j2.properties" %*
And for report generator step I have used this bat as a path for java.
It is dirty, it is ugly, but it works. - For now.
In the meantime, I have created a request for help on the JMeter forum.
Unfortunately, it looks for path "..\bin\log4j2" and if I put it in path "<root>lib\ext" it will look for log config in "..\lib\bin" which doesn't exist.
Unfortunately for that, I will have to change PowerShell scripts - which is the story for a different post. So, until I finish the changes, I have to go with a workaround. I have created lib\bin and put config there.
At this point - the process is working. Both old and new JMeter give the same results for runs.
There are still a few more steps to be done. Fixing PowerShell correctly, propagating changes between agents, and set up environment variables. But those are connected strictly with CI process, so they are outside of this article scope.
There are a few things I have learned while doing this:
- There is too much information noise for JMeter. Finding information for a specific version is hard. This is problematic when there were significant changes between versions.
- There is hardly any information on how to update JMeter to the new version. Hopefully, this case study will help others in that situation -
Open source communities are helpful but moody. - My request got a fast answer but someone asked a similar question years ago, and nobody helped him.
About the author:
Maciej is Senior Automation Quality Engineer at Objectivity.
In his free time, he writes about test automation on his blog: thebrokentest.com