About Engulf, an Ergonomic, Distributed Load Tester
Posted in: Performance Testing   -   August 14, 2013

To an engineer, a bad workflow is a problem to be solved with technology. That’s what was in my head a couple years ago after jotting down a bunch of performance numbers from apache bench and httperf runs in a google docs spreadsheet. The workflow of crunching performance numbers can be arduous and error prone when using traditional tools. The CLI, however, is my home, and it’s not amenable to certain tasks. I’ve usually wound up with dumps of text files, incomplete data in spreadsheets, and data entry errors after all was said and done. It’s a good thing I was about to wind up getting distracted.

While I was having these thoughts about irritation in load testing I’d also been dabbling in Clojure, a JVM language built around advanced concurrency. Basking in confidence singular to the experience of learning a language/tool that makes HARD_PROBLEM_X easy, I was sure I’d have a world class load tester banged out in no time. The timescale was off by quite a number of months, but thank god, for foolish optimism. Blindness led me to build something that addressed a number of the pain points around existing tools. What follows is the list of things I hope I’ve gotten more or less right:

A load tester should be distributed. Not sure if the bottleneck is your testing machine or the target? Spin up n more test machines and be sure you’re saturating the connection.

Managing data should be easy. While I love many of the existing CLI tools, I have no idea how to collate and organize their data. Do I save text files and munge it? Do I need to write a whole database to deal with the data? Do I hand copy stuff to text files? I’ve never really figured it out. Engulf stores data using JSON in SQLite, and makes it available via a JSON REST API.

Administration should be easy. Clojure was a real help here. Your distro just needs its java JRE installed, and a copy of the engulf jar downloaded, and you’re done. No native dependencies whatsoever. engulf.jar is both client and server, a unified codebase. Firewall rules are also simple, all workers connect back to the master meaning you just need to open a couple ports on the master. Don’t want to run it on separate computers? By default it runs in single node mode, bundler the master and workers on a process.

Administration should be even easier: Even lazier than what’s above? http://engulf-project.org has a one-click instant AWS provision button that runs a CloudFormation script that asks you a few questions and builds an arbitrarily sized cluster under your AWS account.

I need a GUI: Visualizations only exist with graphics, and text-mode just doesn’t cut it. While engulf is fully operable via the CLI (just fire up cURL and send off some HTTP requests), it’s even easier to use via your browser.

To sum this all up better, the following diagram illustrates the overall architecture.


Use in The Real World

Real world use cases for a load tester like this are fairly broad. It’s got a wide range of applications, much like any generic load testing tool. I use Engulf frequently myself. The ability to instantly spin up a cluster on amazon is invaluable, because even a large cluster is cheap if only used occasionally. I use it locally as well, where the persistent database of load-test results is great for quickly comparing results. Lastly, there’s a fun feature not found in other load testers, Markov chain request generation, that can take a list of URLs, and issue a load test using access patterns derived from that initial pattern. More on the Markov modeling is available at this blog post I wrote: Using Markov Chains for HTTP Benchmarking.

I’m really excited about two new upcoming features:

  1. External log replay
  2. Scripting support

External log replay is easily the most exciting. Available via the raw API in the master branch on GitHub, it allows the distributed load tester to use a file available either locally on the node, or remotely via HTTP as an input source. The great thing about this is that it streams the data, which means any HTTP endpoint that streams a log of request data can be fed into engulf. It could be used, for instance, to mirror live production requests onto a staging cluster. By adding extra nodes it will even be possible to magnify production traffic by a given factor.

And that, essentially is the why and how of engulf in a nutshell. I hope others find it useful!


Andrew Cholakian is a Software Developer living in Los Angeles CA, working for Pose. He loves hacking on open source software, most of which is up on his GitHub account.

Tags: , , , , , , , , , , , ,

  • http://www.MaxCDN.com/ Chris Ueland / MaxCDN

    great post Andrew. Thank you!

  • http://blog.justindorfman.com jdorfman

    Andrew this is a remarkable tool you have built. I am sure we will be using this at our company to load test new clusters we deploy.

    • andrewvc

      Thanks! If there are any issues just email me or open a github issue.

  • craigforster

    Can Engulf handle more interactive flows, where one request could be used to set up an authenticated session that the subsequent requests use?

    FYI, the Github Usage page points to engulf-project.org, which points back to Github.

    • http://blog.justindorfman.com jdorfman

      @andrewvc:disqus ^^

  • reshefm

    This is a great tool! Thanks for creating it. Small question however – Is there a way to define json body on POST request? I couldn’t find it in the docs…