blitz.io: Benchmarking view performance in CouchDB

March 15, 2011 by kowsik

If you are running CouchDB on an EC2 node with an EBS volume (or otherwise), it’s kinda hard to predict how the view performance will be affected by disk seeks and other criteria. Add complex reduce functions to this mix and you are flying blind. Just in my last blog, we talked about adding parameterization to blitz.io so you can easily test location-aware iPhone and Android apps. What if we could use CouchDB itself to generate fully parameterized tests so you can blitz it?

Parameterization

blitz.io currently supports a few commonly used variables that appear in RESTful requests. Here’s a summary of what we have:

-v:var1 number[-120,120]
-v:var2 alpha[3,7]
-v:var3 list[1,2,abcd,foo,bar,hello,world]

where var* are the variable names. number randomly (seeded so it’s deterministic) generates a number between the given range. alpha randomly generates a string between the given min/max lengths. Finally list acts like an enum and generates one of the specified values.

You can use these variables in any part of the request, including query parameters, headers and POST/PUT content.

Why parameterization?

On pcapr.net we have more than 50 design views and at any given time different views are going to be queried by different users with varying startkey/enkey’s and limit’s. So simply running concurrent HTTP requests on the same view doesn’t reflect the production environment at all. You really need to be able to pass in multiple values to various parts of the query to accurately recreate production workloads.

Auto generating tests from CouchDB

blitz.io itself is fully RESTful and we’ve released a Ruby Gem (uses an API key) so you can integrate blitz into Rake tasks or Heroku’s post-commit hooks. We’ve added a new command to the gem that works like this:

$ gem install blitz
$ blitz couch:fuzz http://localhost:5984 pcapr

You simply point this to a local or remote CouchDB along with the database to use for automatic test generation. What this does is the following:

  • Walk each design document in the database
  • Fetch each view with include_docs and see if it contains map and/or reduce functions
  • Query the first and last row of this view to inspect the keys generated by CouchDB
  • Use these keys (string, array, boolean, numbers) to generate parameterized blitz tests

The end result looks like this (we’ve folded the lines for clarity):

-------------- processing pcapr... ----------------------
  analyzing pcaps/by_created_at...
  analyzing pcaps/by_path...
  analyzing pcaps/by_service...
  analyzing pcaps/by_keyword...
  analyzing pcaps/statistics...
  analyzing pcaps/indexed...
  analyzing pcaps/by_filename...
  analyzing pcaps/by_inode...
  analyzing pcaps/by_directory...
  analyzing pcaps/by_status...
  analyzing pcaps/queued...

-v:sk number[1298940724000,1299791588000]
-v:ek number[1298940724000,1299791588000]
-v:l number[1,10]
-v:id [true,false] 

http://dell-5:5984/pcapr/_design/pcaps/_view/by_created_at?

    startkey=#{sk}&endkey=#{ek}&include_docs=#{id}&limit=#{l}

-v:l number[1,10]
-v:gl number[1,5] 

http://dell-5:5984/pcapr/_design/pcaps/_view/by_path?

    group=true&limit=#{l}&group_level=#{gl}

-v:sk alpha[4,12]
-v:ek alpha[4,12]
-v:l number[1,10] 

http://dell-5:5984/pcapr/_design/pcaps/_view/statistics?

    group=true&startkey=%22by#{sk}%22&endkey=%22sez#{ek}%22&limit=#{l}

...

Blitzing

Simply copy/paste the line into the blitz.io, add a –pattern 100-100:60 and run. What this does is generate 100 concurrent requests to CouchDB over a period of 60 seconds, except all the query parameters will be different and unique for each concurrent request. And you get a pretty little graph like this:

Only downside is your CouchDB needs to be on the public cloud. Hopefully this will help you understand view performance in CouchDB and optimize your reduce functions so you get scale out your application. Incidentally, there’s a term for this automatic test generation. It’s called Fuzzing! :-)

If you think you can do something similar for Sinatra and Rails apps using rspec, routes and mocks, please generate a pull request!

blitz.io: Making load and performance testing a fun sport!

Categories: Announcements, Blitz, Cloud, CouchDB, Fuzzing, Testing
5,563 Comments
Bookmark and Share