Friday, January 29, 2010

Poor Beanshell Performance and Custom Functions for JMeter

I'm building a relatively complex JMeter test plan to simulate load on the Kikini website. As soon as you need to do anything remotely complex, you exceed the capability of the built-in JMeter configuration elements and functions. The initial version of my test plan therefore used the BeanShell capability, which allowed me to do relatively complex things in a familiar language (BeanShell is essentially interpreted Java).

All fine and good until we need to run tests longer than 10 minutes or with more than 10 threads. An issue in BeanShell causes massive slowdowns if used inside loops (eg, inside a sampler), which in fact was what I was doing. When I worked around the issue by resetting the interpreter on each call, I found that JMeter was spending so much time processing BeanShell code that it couldn't effectively scale up to more than about 10 threads. The bottom line is that BeanShell is unfit for use if it must be called repeatedly in a JMeter test.

The only way I could find to get the complex behavior I want without compromising performance was to implement my own JMeter function. JMeter offers a number of simple functions out-of-the-box. Although JMeter isn't really an API, it does have a Function interface which you could implement. Then from inside any test element, you can call your function:

${__myFunction(arg1, arg2)}

And you'll get back a string that is the result of your function. Before we get to function class itself, there is some background to discuss.

First, JMeter isn't an API. But with a little bit of work, you can program against it. If you download the JMeter binary distribution, you can extract ApacheJMeter_core.jar. This JAR contains the interfaces you'll code against.

Second, you need a way to get your custom function onto JMeter's classpath. You can set the search_paths system property, and JMeter will find it. This is great because then you do not have to modify the JMeter distribution to use your custom functions.

Once you're ready with your custom JAR, you can invoke JMeter:

jmeter -Jsearch_paths=/path/to/yourfunction.jar

Alright, on to the code. This is a skeleton (please ignore the naming) which will simply return Array.toString() on the arguments you give:

package com.kikini.perf.jmeter.functions;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.List;

import org.apache.jmeter.engine.util.CompoundVariable;
import org.apache.jmeter.functions.AbstractFunction;
import org.apache.jmeter.functions.InvalidVariableException;
import org.apache.jmeter.samplers.SampleResult;
import org.apache.jmeter.samplers.Sampler;

public class MaskUserIDFunction extends AbstractFunction {

    private static final List<String> DESC = Arrays.asList("uid_to_mask");
    private static final String KEY = "__maskUserID";

    private List<CompoundVariable> parameters = Collections.emptyList();

    public String execute(SampleResult arg0, Sampler arg1) throws InvalidVariableException {
        List<String> resolvedArgs = new ArrayList<String>(parameters.size());
        for (CompoundVariable parameter : parameters) {
        // TODO: mask the user ID in resolvedArgs.get(0). For demo purposes,
        // just return the arguments given.
        return resolvedArgs.toString();

    public String getReferenceKey() {
        return KEY;

    public void setParameters(Collection arg0) throws InvalidVariableException {
        parameters = new ArrayList<CompoundVariable>(arg0);

    public List<String> getArgumentDesc() {
        return DESC;


There are a few crucial things to note here. The package name contains ".functions". That is a requirement, otherwise your function will not be recognized by JMeter. Notice that the type of the arguments is CompoundVariable. You must call execute() on them to resolve them to a String.

Otherwise this is relatively straightforward. Now I can call my function from inside a sampler:

And it will return the correct results:

So, how do Java functions perform versus the BeanShell functions? My test plan had about 10 samplers, most of which used BeanShell before, but now use native Java functions. My dedicated JMeter machine is a dual-core system with 2GB of RAM.

Before: JMeter maxed out at ~45 requests per second, 90%+ CPU usage
After: Generates 150+ requests per second with 2-3% CPU usage

Huge win! I don't actually know what the limit is now but I'm guessing I could get thousands of requests per second now.

Sunday, January 24, 2010

Releasing simpledb-appender as open source

I've released the SimpleDB appender I wrote as open source under the Apache 2.0 License. The project is hosted here:

The purpose of this project is to allow Java applications using the SLF4J API with Logback to write logs to Amazon SimpleDB. This allows centralization of the logs, and opens powerful querying capabilities. Also scripts and tools are included so that even non-Java applications can have their stdout/stderr logged to SimpleDB as well.

The project is tested and works well. Developers familiar with SLF4J should have no problem integrating it into their apps. The documentation for using it as a tool for non-Java applications is a little weak but I have a demo shell script that should at least get folks started.

Let me know how it works for you!

Thursday, January 14, 2010

Amazon Web Services Expanding into Asia

Last year, I privately speculated that having launched datacenters in the Eastern US and Western Europe, the next obvious locations for Amazon Web Services (AWS) would be the Western US and Asia. In December 2009, AWS announced availability zones in Northern California.

What I didn't realize until today was the AWS actually announced their intentions to expand into Asia back in November 2009. Multiple availability zones will be available in Singapore in the first half of 2010.

Singapore does make some sense as a location. A glance at the map (source: reveals that Singapore is pretty central, located roughly equidistant from China, India, and Australia. So if AWS is persuing a strategy to minimize the average global latency, it is probably a good choice. It also offers a relatively stable political and economic environment, though there is some political risk to locating yourself in an authoritarian country.

But when I first thought about a datacenter in Asia, my thought would have been hosting it in Korea. Korea is one of the most connected (in the data networking sense) countries on Earth, and is in close proximity to the other two most important markets in Asia: China and Japan. Korea is a very stable political and economic environment, and doesn't have the significant political risk associated with hosting in China or the less significant risk of Singapore. Latency from Korea to China and Japan is very low. I imagine the cost of running a datacenter in Korea is not much more expensive than Singapore, given that living standards are comparable.

Still, I can't complain. Hosting in Singapore will allow a better web experience for users throughout Asia. I hope to see AWS continue expanding geographically.