Functional and Fuzz Testing Proxies and Load-Balancers

May 19, 2010 by kowsik

Using Mu Studio, we recently enabled one of our customers to do both functional and fuzz testing of their proxy/load-balancer (P/LB). This P/LB supports SSL termination, IPv6, Caching, Compression, WAN Acceleration & Optimization and a plethora of really cool features that enable high-volume cloud apps and web sites. What I want to talk about is our approach to testing complex products/deployments like these from both a functional and fuzz/security/resiliency testing perspective.

The challenge

Traditional bit-blasting solutions have the innate problem of customizing/generating traffic to specific scenarios that can be used for testing. While these can pump out large gobs of mindless pre-programmed packets, they completely fail at testing a P/LB. Most testing tools come in two varieties: They can either be both the client and the server (read my last blog on using dumb packet-replay for testing modern firewalls) or they are one-armed and can test a live server through the proxy.

With compression and SSL termination, the P/LB maps HTTPS flows into HTTP flows (as one example) so you can use commodity servers to fan-out inbound requests. This is typically how you scale high-volume web sites because you have the P/LB dealing with CPU-intensive tasks and distributing the load.

Here are a few specific challenges in testing P/LB’s:

  • Correlating how an HTTPS flow was transformed into an HTTP flow and asserting that the transformation was legitimate
  • Testing browser-specific caching tweaks
  • Busting the cache by generating large number of URL’s and validating them on the other side
  • Fuzz testing application traffic sent over HTTPS

What did we do?

We initiated a simple browser request through the P/LB to a server behind it and captured the traffic (tcpdump) on both interfaces of the P/LB. We then took these pcaps and imported them into Mu Studio which automagically converts them into MuSL (Mu Scenario Language). MuSL is a OSI layer-agnostic, abstract representation of messages exchanged between multiple hosts over different transports. This is much like BPEL for web services. MuSL contains message descriptions on the send side and assertions that you can add on the receive side. Because it’s an abstract definition, you can see below the call-graph that we automatically generated from this scenario.

plb-call-flow.png

Parametrization

Data-driven Testing is a powerful paradigm for running through large sets of data through a scenario which makes the tester’s job super easy. Mu Studio allows you to parametrize the generated MuSL so you can very easily create options for the URL, the User-Agent, Status Code and so on. You can then bind these to a spreadsheet full of values to be used in testing.

If you look at the call-graph, you will notice that the inbound request for /url0.html is getting mapped to /index.html to the server. These types of transformations, while common with P/LB’s, are fairly difficult to test. The test tool needs to be both the Browser and the Server sending the right message through the P/LB and validating the received message on the other side.

Now if you have a large set of URL’s, it gets pretty cumbersome to enter each one of them in a spreadsheet. So using the Popcorn technology that we open-sourced a while back, we used Mu functions to automatically generate large sets of URL’s that we can use for testing.

Each line in the spreadsheet has the requested URL from the Browser, the transformed URL on the Server-side as well as the final expected status code on the Browser-side.

Request URL Received URL Final Status
/url0.html /index.html 200
/nope.html /nope.html 404
Mu.list(‘a.html’, ‘b.html’, ‘c.html’, ‘d.html’) /404page.html 404

Testing

When it’s time to test, you bind the abstract host definition in MuSL to either the Mu interface or an external IP address. In this case, we want Mu to be the Browser on one side acting as a client and the Server on the other side active as a server. Once this binding is specified, Mu Studio takes on the specified roles for the hosts and automatically switches from being a client or a server.

The exact same scenario can be used to test servers or to test the P/LB by being both the pitcher and the catcher!

Fuzzing

Well, this part’s the easiest. By the time you finish importing the pcaps into Mu Studio, you automatically get fuzz tests for all messages sent and received in the abstract scenario. This means you get server-side fuzzing for free!. Here’s a screenshot of the fuzz-tests automatically generated for this scenario:

plb-fuzz-tests.png

That’s over 6000+ test cases and all you had to do was capture some packets!

And while the captured packets were over IPv4, running the scenario over IPv6 is a single drop-down away. Simple, the way testing should be.

Summary

Scenario-based testing is a powerful new approach to testing complex products, technologies and deployments. Because it uses captured packets as the basis for generating test cases, it’s unique and most relevant for what you want to test. Not some contrived RFC-based tests that are given to you by the test tool vendor.

If you want to learn more, check out Mu Studio!

Categories: Fuzzing, IPv6, Studio, Testing
738 Comments
Bookmark and Share