Friday, November 13, 2009

Using Maven Chronos Without an External JMeter Install

Performance is one of the things we're really focused on at Kikini. But we want to stay focused on actually improving performance, and not spending a lot of cycles making manual measurements and interpreting logs. JMeter is probably the best open-source tool out there for measuring performance of a web application. I designed a JMeter test plan to simulate users visiting our site. Unfortunately while JMeter is great at making measurements, it stops short of data analysis and reporting.

Ideally we would like to get perf reports out of every build, which means we would like to do reporting as part of our Maven build, with results available as easily readable charts on our build server. The top hit you're likely to get from searching for "maven jmeter" is the awful JMeterMavenPlugin. I say awful because it wasn't easy to integrate, and if you look at the source code it's obvious that the project was done in spare time. There are a number of comments in the source like "this mess is necessary because..." which makes me think the whole thing is poorly designed, and if you search around you will indeed find that there are a number of problems people have encountered trying to use it. Finally, the output from the plugin is just the simple JMeter log, and not the reports I'd like.

All the way down in the middle of the second page of the Google results I found this gem: chronos-maven-plugin. Not only does this look like a well-designed and well-executed project, it produces wonderful HTML reports, perfect for plugging into our build server! This is a snippet of what the Chronos output looks like:



The only downside is that the Chronos plugin requires an external install of JMeter, which kind of defeats the whole purpose of Maven. Fortunately, inspired by an Atlassian post, I worked out a way to use the Chronos plugin without making JMeter a manual install by using the maven-dependency-plugin. First I deployed the JMeter ZIP file as an artifact on our Artifcatory repository:

<?xml version="1.0" encoding="UTF-8"?>
<project>
  <modelVersion>4.0.0</modelVersion>
  <groupId>org.apache.jmeter</groupId>
  <artifactId>jmeter</artifactId>
  <version>2.3.4</version>
  <packaging>zip</packaging>
  <description>Artifactory auto generated POM</description>
</project>


In my POM, I set jmeter.home to the location that we'll be unpacking JMeter into:

<properties>
  <jmeter-version>2.3.4</jmeter-version>
  <jmeter.home>${project.build.directory}/jakarta-jmeter-${jmeter-version}</jmeter.home>
</properties>


Next I use the dependency plugin in the pre-integration-test step to unpack JMeter into the target folder:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-dependency-plugin</artifactId>
  <version>2.1</version>
  <executions>
    <execution>
      <id>unpack-jmeter</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>unpack</goal>
      </goals>
      <configuration>
        <artifactItems>
          <artifactItem>
            <groupId>org.apache.jmeter</groupId>
            <artifactId>jmeter</artifactId>
            <version>${jmeter-version}</version>
            <type>zip</type>
          </artifactItem>
        </artifactItems>
        <outputDirectory>${project.build.directory}</outputDirectory>
      </configuration>
    </execution>
  </executions>
</plugin>


Finally I configure Chronos to run:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>chronos-maven-plugin</artifactId>
  <version>1.0-SNAPSHOT</version>
  <configuration>
    <input>${basedir}/src/test/jmeter/UserSession.jmx</input>
  </configuration>
  <executions>
    <execution>
      <goals>
        <goal>jmeter</goal>
        <goal>savehistory</goal>
      </goals>
    </execution>
  </executions>
</plugin>


Bingo. Now anyone running our build can get the JMeter performance reports with nothing more complex than running "mvn verify chronos:report".

13 comments:

  1. Hi
    Could you please help me? I have ready jmeter report: aggregate_jmeter.jtl and I want to build graphs with maven-chronos-plugin. What I need to do?
    ps: I have pure knowledge of java
    It would be great if you-ll email me: koldaevav@gmail.com

    ReplyDelete
  2. Maven runs Chronos, and Chronos runs Jmeter. It's meant to take your Jmeter test plan (your .jmx file), and run it and generate the graphs. It doesn't take a Jmeter .jtl file as input. If you're not already using Maven then Chronos won't be much help to you.

    ReplyDelete
  3. Hello,
    Could you please share complete examples of needed Maven's xmls.
    I'm very nub in Maven, but I need continuous testing with jMeter, even without builds of the application

    Thanks!

    ReplyDelete
  4. hi Gabe, did you uploaded the complete jmeter.zip File to your maven repository(~18MB)? The zip contains a directories/files not needed by JMeter e.g. "docs". I'm not sure which files have to remain in the zip. On http://wiki.apache.org/jakarta-jmeter/JMeterMavenPlugin I found a JMeter.jar and a pom but this is for JMeter 2.2

    ReplyDelete
  5. Martin -- more than just JMeter.jar is required because there are a number of dependencies inside the ZIP file. I like just putting the ZIP in the repository because then it makes the process of upgrading to a new version of JMeter very simple -- just upload the new ZIP and update the version in the pom.xml

    ReplyDelete
  6. Hi Gabe

    How customizable have you found Chronos, if at all? If you want to generate custom reports, with more than the bubbled up pieces of information (such as per sample or group of sample records etc), is Chronos a good tool? I did not see any such features listed in its doc

    Thanks

    ReplyDelete
  7. The output report is pretty much fixed in code. You can tweak some very minor elements of the report, such as whether to include garbage collection logs.

    Chronos is great for getting measurements and a simple, readable report out of the build without too much work. If you need more, you need to download the source and modify it to suit your needs. I made some minor modifications in this way. But my company is probably going to move to an in-house reporting system that isn't tied to JMeter as a harness and lets us generate the graphs exactly how we want 'em.

    ReplyDelete
  8. Thanks Gabe. That is probably the direction we are headed in too. We will probably use JMeter for logging only, and use other tools or in-house tools to generate reports

    ReplyDelete
  9. Since we may open-source our reporting system, I'm curious to see what scenarios other people would be interested in. What sort of customization are you looking to do? What information do you want to extract beyond the simple graphs/data chronos provides?

    Thanks!

    ReplyDelete
  10. Sorry for not posting back on your last question. I have not yet figured out the reporting needs exactly.

    However, I tried the method you mentioned, and it takes loooooooooooooooooong time (of the order of 4-5 minutes) just to unpack the zip into my test directory. Did you see this issue too?

    If this is the case, I may have to resort to having local jmeter installs, and provide the location using the jmeter.home property as a command line (system property) parameter.

    Thanks

    ReplyDelete
  11. No, it's quite fast for me. Are you sure it's the unpack step that's slow? Maven isn't doing other things like downloading dependencies?

    In my logs the time between the "Unpacking" step and the next one is 4 seconds.

    You might try running mvn with --debug to figure out why it's so slow.

    ReplyDelete
  12. Out of interest have you had a look at the Jmeter Maven Plugin recently? Quite a bit of development effort has gone into it since you last looked at it and there is also a JMeter Analysis Maven Plugin available now as well.

    I'd be interested to know if you opinion has changed:

    https://github.com/Ronnie76er/jmeter-maven-plugin

    ReplyDelete
  13. I just had to say that this saved me hours of work. Thanks Gabe!

    ReplyDelete