Automated Functional Testing with JMeter

Jmeter bills itself more as a load testing tool, but we’ve used it extensively for functional testing as well. For us, functional testing amounts to pretending you are a (single) user hitting the application with various requests on the way to performing a task. For example, in an ecommerce application, adding a product to your cart, going through checkout and placing an order. In other words, making sure the thing works the way it’s supposed to.

Testing of course is vital, if not central, to development. That said, there is a point of diminishing returns where it takes up more time than its worth. The line is crossed at different points depending on the situation: how mission critical the function is, what would happen if something goes wrong, the client’s tolerance for risk and instability, etc. We rarely do unit testing, which I know will cause conniptions in some. But we have to do functional testing. Someone has to sit down and see if the thing works! JMeter allows us to automate part of this process, making our testing more reliable and less time-consuming.

We have a “place an order” functional test set up to run on an hourly basis for every client we work with. We run it not only during development, but continuously on their live sites. The test script initiates HTTP requests for the client’s website, adds an item to its cart, and goes through the checkout process. Some clients have it skip the last step of checkout so an order is not actually placed, other clients have it actually place a small order. The script is tailored for each client based on their specific configuration (for example, how they have configured the flow of screens during checkout).

In addition, we have expanded the hourly script for some clients to test custom functions we’ve developed for them. In one case, the script logs in as an administrative user and goes through 90% of the functions we developed for them to assist in processing the order. For them this includes creating FedEx shipping labels, sending out emails, changing the order’s status, canceling the order, charging refunds, and so on.

In this light, the script operates as a continuous regression test of the system, not to mention a monitor that the site is up in the first place. If anything is introduced that breaks something, there’s a decent chance the script will detect it and email us. It also serves to let us know if any of the services we integrate with are down, such as FedEx’s web services, or our Authorize.net integration.

Our Basic, No-Frills Test Script

So I’d thought I’d share our generic SoftSlate Commerce JMeter functional test script. Feel free to run it. Although, you will be placing a test order on our demo store, so please don’t go crazy and DOS us. (This is functional test, not a load test.)

To view this the best way is to download JMeter, unzip it, and run

bin/jmeter &

on unix or

bin/jmeter.bat

on Windows. A GUI will pop up, and you can then open up the above .jmx file using File -> Open to look at it. The .jmx file is simply an XML file, which can edit via some other editor directly too.

I will leave it up you to explore the script and point you to the JMeter user manual for reference. But I should just point out a couple things. First, to run the script, it’s useful to highlight the “View Results Tree” so you can watch the action as it unfolds:

Second, note the two “Assertions” in the script. One checks to make sure the footer.jsp file was included in every request. That tells you the response got through to the bottom of the page, which is a good thing. The other tests to make sure no errors were displayed during the course of the test, also a good thing:

Automate Me Please

Now the fun part is to attach this test to a script and run it with a cron so you can rest easier that your website is up and running at all hours (or at least know when it wasn’t). I don’t have directions for doing this on a Windows platform, sorry. 90% of our clients are on some flavor of Linux. With that said, here’s a real basic shell script that will invoke the JMeter test script:

#! /bin/sh
/usr/local/jmeter/bin/jmeter-dt -n
                           -t /usr/local/jmeter/scripts/softslate.com.jmx
                           -l /usr/local/jmeter/scripts/softslate.comLog.jtl

Essentially, you put the above into a file and put that file into /etc/cron.hourly, and you’re off and running with an hourly, automated functional test. For all the options, check out out Non-GUI mode in the JMeter docs. In short, -n signifies running JMeter as a command, -t points to the .jmx script and -l points to the log file.

You might notice I tweaked the default jmeter command and named my version jmeter-dt. The only reason I did this was that by default the jmeter command’s memory parameters are too low for us. Here is a diff of those relevant changes:

[dtobey@centos5 bin]$ diff jmeter jmeter-dt
< HEAP="-Xms256m -Xmx256m"
---
> HEAP="-Xms16m -Xmx32m"
49c42
< NEW="-XX:NewSize=128m -XX:MaxNewSize=128m"
---
> NEW="-XX:NewSize=8m -XX:MaxNewSize=8m"
78c71
< PERM="-XX:PermSize=64m -XX:MaxPermSize=64m"
---
> PERM="-XX:PermSize=16m -XX:MaxPermSize=16m"
87c82
< java $JVM_ARGS $ARGS -jar `dirname $0`/ApacheJMeter.jar "$@"
---
> /usr/java/j2sdk/bin/java $ARGS -jar `dirname $0`/ApacheJMeter.jar "$@"

One more missing piece is email notification of all failures. Fortunately JMeter has a thing called a Mailer Visualizer. Set up is pretty easy; here is a screenshot:

(The above .jmx file has this in there, but just disabled.)

If I have one complaint about the Mailer Visualizer, it’s that it only sends you the URL of the page that failed, not anything about the Assertion that failed. Alas, JMeter is all open source, so I should just dive into the code and add the feature. Call it laziness, I don’t mind.

I hope that was helpful. Thanks for reading!

Update – Getting JMeter to Deal with HTTPS Requests

One thing I just noticed is I uncommented a couple lines in bin/jmeter.properties related to SSL. You may need to try this in order to get JMeter to handle making HTTPS requests. Here is a diff of the relevant changes:

[dtobey@centos5 bin]$ diff jmeter.properties jmeter.properties.orig
57c57
< ssl.provider=com.sun.net.ssl.internal.ssl.Provider
---
> #ssl.provider=com.sun.net.ssl.internal.ssl.Provider
63c63
< javax.net.ssl.keyStore=/usr/java/jdk1.5.0_02/jre/lib/security/cacerts
---
> #javax.net.ssl.keyStore=/usr/java/jdk1.5.0_02/jre/lib/security/cacerts
66c66
< javax.net.ssl.keyStorePassword=changeit
---
> #javax.net.ssl.keyStorePassword=changeit

Enjoy!