Friday, January 29, 2010

Poor Beanshell Performance and Custom Functions for JMeter

I'm building a relatively complex JMeter test plan to simulate load on the Kikini website. As soon as you need to do anything remotely complex, you exceed the capability of the built-in JMeter configuration elements and functions. The initial version of my test plan therefore used the BeanShell capability, which allowed me to do relatively complex things in a familiar language (BeanShell is essentially interpreted Java).

All fine and good until we need to run tests longer than 10 minutes or with more than 10 threads. An issue in BeanShell causes massive slowdowns if used inside loops (eg, inside a sampler), which in fact was what I was doing. When I worked around the issue by resetting the interpreter on each call, I found that JMeter was spending so much time processing BeanShell code that it couldn't effectively scale up to more than about 10 threads. The bottom line is that BeanShell is unfit for use if it must be called repeatedly in a JMeter test.

The only way I could find to get the complex behavior I want without compromising performance was to implement my own JMeter function. JMeter offers a number of simple functions out-of-the-box. Although JMeter isn't really an API, it does have a Function interface which you could implement. Then from inside any test element, you can call your function:

${__myFunction(arg1, arg2)}

And you'll get back a string that is the result of your function. Before we get to function class itself, there is some background to discuss.

First, JMeter isn't an API. But with a little bit of work, you can program against it. If you download the JMeter binary distribution, you can extract ApacheJMeter_core.jar. This JAR contains the interfaces you'll code against.

Second, you need a way to get your custom function onto JMeter's classpath. You can set the search_paths system property, and JMeter will find it. This is great because then you do not have to modify the JMeter distribution to use your custom functions.

Once you're ready with your custom JAR, you can invoke JMeter:

jmeter -Jsearch_paths=/path/to/yourfunction.jar

Alright, on to the code. This is a skeleton (please ignore the naming) which will simply return Array.toString() on the arguments you give:

package com.kikini.perf.jmeter.functions;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.List;

import org.apache.jmeter.engine.util.CompoundVariable;
import org.apache.jmeter.functions.AbstractFunction;
import org.apache.jmeter.functions.InvalidVariableException;
import org.apache.jmeter.samplers.SampleResult;
import org.apache.jmeter.samplers.Sampler;

public class MaskUserIDFunction extends AbstractFunction {

    private static final List<String> DESC = Arrays.asList("uid_to_mask");
    private static final String KEY = "__maskUserID";

    private List<CompoundVariable> parameters = Collections.emptyList();

    @Override
    public String execute(SampleResult arg0, Sampler arg1) throws InvalidVariableException {
        List<String> resolvedArgs = new ArrayList<String>(parameters.size());
        for (CompoundVariable parameter : parameters) {
            resolvedArgs.add(parameter.execute());
        }
        // TODO: mask the user ID in resolvedArgs.get(0). For demo purposes,
        // just return the arguments given.
        return resolvedArgs.toString();
    }

    @Override
    public String getReferenceKey() {
        return KEY;
    }

    @SuppressWarnings("unchecked")
    @Override
    public void setParameters(Collection arg0) throws InvalidVariableException {
        parameters = new ArrayList<CompoundVariable>(arg0);
    }

    @Override
    public List<String> getArgumentDesc() {
        return DESC;
    }

}

There are a few crucial things to note here. The package name contains ".functions". That is a requirement, otherwise your function will not be recognized by JMeter. Notice that the type of the arguments is CompoundVariable. You must call execute() on them to resolve them to a String.

Otherwise this is relatively straightforward. Now I can call my function from inside a sampler:



And it will return the correct results:


So, how do Java functions perform versus the BeanShell functions? My test plan had about 10 samplers, most of which used BeanShell before, but now use native Java functions. My dedicated JMeter machine is a dual-core system with 2GB of RAM.

Before: JMeter maxed out at ~45 requests per second, 90%+ CPU usage
After: Generates 150+ requests per second with 2-3% CPU usage

Huge win! I don't actually know what the limit is now but I'm guessing I could get thousands of requests per second now.

11 comments:

  1. This comment has been removed by a blog administrator.

    ReplyDelete
  2. Thanks for posting this! It's an excellent description of the process. I ended up just placing my function JAR in ...lib/ext because I was too lazy to determine the correct Windows path syntax for search_paths :)

    Also, my custom function returns a more or less random string, so I had no need for input parameter processing.

    Appreciate your work!

    ReplyDelete
  3. Searched for "JMeter Custom function not called" and found your webpage. Your namespace(.functions) note was my problem. I Appreciate the help!

    ReplyDelete
  4. Thank you for the post. Do you know if there is anyway to pass a variable to the custom function?

    ${__maskUserID(${var_1}, ${var_2})}

    ReplyDelete
  5. Yes, passing in variables works fine. The call to parameter.execute() on line 26 resolves those variables.

    ReplyDelete
  6. Thank you for the reply. I found my issue. My initial thought was to get the variable from the JMeterVariables returned from getVariables() instead of passing the variable around. My problem I was doing this in setParameters() and my variable wasn't in the set. Moving this logic to execute() worked. I assume JMeter is doing more with the context in between setParameters() and execute().

    Thank you again for a great post and the quick reply.

    ReplyDelete
  7. Great post :) I was wondering how to define custom functions in jmeter and here's the answer.
    Thx gabe

    ReplyDelete
  8. Thanks Gabe! This is what I was looking for since many days! :)

    ReplyDelete
  9. I wasted a bunch of time trying to get JMeter to see my custom function. Turns out that problem was my jar file. I just did a "jar cvf CustomFunction.jar myfunc.class", but JMeter wanted the jar file path to match the package, so moving myfunc.class to com/myco/functions/test and "jar cvf CustomFunction.jar com" solved it right away.

    ReplyDelete
  10. Do you think you could re-post the images of where you use your function in the sampler? I do not see the sampler images below the code.

    ReplyDelete
  11. Unfortunately the S3 bucket belonged to my old company and I guess they deleted it :(

    ReplyDelete