" title="Mu Dynamics Blog">Mu Dynamics Blog - insites into vulnerabilities and secure coding practices

I felt a Déjàvu moment today when one of our customers came to us asking if we can help them test Outlook Exchange traffic through their firewall with ALG and NAT turned on. They had tried to re-purpose bit-blasters, load generators, open-source and commercial packet replay tools only to find that nothing was working. Way back when I was building the IDP at OneSecure, my pre-screen interview question was this:

If you only had an [ any, any, tcp/21, allow ] rule in your packet filter, why wouldn’t FTP uploads/downloads work?

The answer of course, a decade after the question was posed, is now pretty simple. Most modern security gateways, UTM’s, firewalls, IPS’ and load balancers all inspect application payloads. They look for IP addresses and port numbers embedded in the application payload in order to open up temporary pin-holes. This helps in two ways:

• Allow traffic on ephemeral ports assuming that the control flow was allowed
• Perform Network Address Translation or NAT to rewrite IP and ports on the flly

This is all old hat. But some things never change while others do. The number of new network devices that all do some level of application identification and deep packet inspection has exploded. But it looks like the testing tools that were oh-so-good at pushing a bazillion packets/second never quite caught up with the changes. These tools were awesome at pushing large gobs of mindless packets through a router that just decremented the TTL. But the moment ALGs got in the way all hell broke loose.

More importantly these tools are limited in what kind of test traffic they can generate. They have to be programmed by the vendor or else you are SOL.

Let’s walk through a concrete example. The Outlook Exchange traffic below uses something called EPM or Endpoint Mapper. This is very similar to portmap in Sun RPC protocols where a service runs on a well-known port to which you can ask where a named service (using an UUID or a program number) is running. Once the EPM returns you the ephemeral port number, you can then connect to it. Here’s the relevant information from Wireshark:

Frame #8 is the Map Request and Frame #9 (shown in detail) is the Map Response that contains the DCERPC Protocol Towers. These towers tell the client the IP address and the port numbers where the requested service is running. Finally Frame #10 shows the second TCP connection opened by the client to port 31337 (the ephemeral port).

What do you think the firewall ALG is doing?

It’s inspecting the protocol towers and opening up a pin-hole for TCP/31337 from 192.168.1.1 to 192.168.1.2!

Now blind packet replay (especially with concurrency/load) is most likely going to change the IP addresses and the port numbers of the TCP/IP headers, but the firewall code is looking at the pin-hole information which means when the second TCP connection hits the firewall, it’s just going to reject the SYN.

So all those bazillian packets get routed to /dev/null. What a waste of time!

Testing with Mu Studio

With Mu Studio, we took a radically different approach to generating test cases from packet captures. Here’s what happens when you upload a pcap into Mu Studio:

• Using the same xtractr technology we launched a while back, we classify flows and allow the user to pick one or more flows to import
• We then normalize, parametrize and transform the pcap into something called MuSL (Mu Scenario Language)
• Using a number of rules and heuristics, we auto markup MuSL to identify ephemeral ports and ip addresses in the application payload. This is exactly what ALG does!
• This step also includes marking lengths and checksums and all the other gory things found in protocols

Now this is cool, ‘cos we can automatically replace IP addresses and ports inside the messages to reflect the testbed. Because all the lengths and checksums are marked-up, changing one part of the packet automatically propagates across the rest recomputing lengths, offsets and all other dependent fields. TCP messages are sent over real TCP sockets, not some hacked up TCP stack.

This is not that different from what Mercury did with the correlation in their LoadRunner product.

But what’s really going to bake your noodle later on is once the scenario is generated you can execute it against a real Exchange server where the ports are all ephemeral and everything just works! And don’t forget that you started all this from a simple little packet capture. And want to run the same test over IPv6? Simply change the transport mode and voila’. All embedded IPv4 addresses automatically switch to IPv6. That simple!

Because of the way MuSL works, here’s the automatic call graph generated by the product when we uploaded the pcap above:

The key aspect of Mu Studio is that your firewall with ALG, NAT and all other application identification turned on will never know that the traffic is generated by Mu Studio. It thinks it’s Outlook, which is exactly what a test tool needs to do.

Fuzzing with Mu Studio

As I mentioned in my CanSecWest presentation a couple of years ago, you don’t build a fuzzer. You build an engine that can serialize normal messages in an alternate, intelligent way. Since the transformation of a pcap into MuSL is fully field-type-aware, here’s the set of test cases that Mu automatically generated from the pcap above:

Each variant is a test-group which means by simply uploading a pcap into Mu Studio, we just generated 10,000+ fuzz-tests! Have a different scenario with a different pcap for those 400+ protocols on pcapr? No problem, we can handle it.

So next time you are testing any modern firewall with your favorite test tool, try it with ALG and NAT. If all packets end up in /dev/null, then you just wasted a lot of time and $$$. Try Mu Studio instead.