We're currently using The Grinder for our Sync load tests at Mozilla
Services. Grinder will let you write the tests using Jython and run
a distributed load test. Despite the Java and Jython environment layer
which is always painful to set up, Grinder works well.
But do not expect it to produce nice reports, you're on your own for
this. You need to work on the raw results to produce nice diagrams. Do
not expect either to perform some operations on the test nodes before
and after the distributed load test is executed. Tests are driven via an
UI and while it's probably possible to script them via Java... it's
Java. And well, the feedback when the tests are running is nearly
The distributed load system of my dreams must:
- let me run arbitrary commands on every node before and after it runs the test. And retrieve results then merge them in a consolidated result. - let me write complex unittest-based functional tests that I can also use to check the health of a server. - provide basic reporting features - be written in Python ;)
Oh.. hold on.. I have it : Funkload and Fabric.
Funkload let you write complex functional tests, does load testing
with complex cycles and produces nice reports.
And then there's Fabric, which allows you to run commands via SSH
on a pool of servers.
Funkload is not distributed but it's fairly simple to drive it via
Fabric. And when all nodes have finished the execution --whom you can
watch via the Fabric stdout-- you can get them back and merge them in a
So basically, the fabric script looks like this:
from fabric.api import run, get, env def runtest(): run('fl-run-bench testmodule TestClass.test_function') file = '/path/to/results.xml' get(file, env.host_string + '-results.xml')
This will run the command on the distant server, then download the xml
file. All through SSH. It can be launched on every server with:
$ fab -H the.server.org runtest
Fabric does not provide parallel execution yet, so you have to create a
batch to run the script on every server in parallel processes. But this
feature should be included in Fabric 1.1. There's a branch for this:
Then you can merge the files in a single XML file like this:
from funkload.MergeResultFiles import MergeResultFiles import os def merge_results(): files =  for file_ in os.listdir(HERE): if not file_.endswith('-results.xml'): continue files.append(file_) MergeResultFiles(files, 'results.xml')
And finally, create an HTML output:
$ bin/fl-build-report --html -o html results.xml
Granted, there's no control of the nodes during the tests, that feature
would require a small TCP server that drives Funkload on each server.
And that would obsoletes the need for Fabric I guess. But for setting up
a quick distributed test, I am good with SSH for now.
I should also mention Benchmaster, which do something similar but
seems complex to configure compared to a simple Fabric script.
Funbric ? :D