Visualizing Application Flows with xtractr

September 30, 2010 by kowsik

If you haven’t checked out xtractr already, you should! It’s a RESTful server that indexes large packet captures for the purposes of forensics, data extraction, reporting, etc. While xtractr can generate all sorts of cool reports and charts, they don’t quite capture the dynamic essence of the network. Users come and go, they tweet, machines send queued emails, phone calls fly around, files get transferred. Static reports and visualizations (Top Talkers anyone?) just don’t do justice to this flurry of activity that happens on a network.

Quick introduction to Flows

During the indexing process, xtractr classifies and groups related packets into flows. The simplest flow is a TCP connection which starts off at some point with a number of messages exchanged and then gets closed. By assigning all packets that belong to this TCP connection with the same flow identifier, xtractr can very quickly report on things like connections/second. This higher order information is lost when looking at packets on an individual basis.

Here’s an example of an FTP session that illustrates the point:

So in addition to the usual suspects like src/dst IP, src/dst ports, each flow in the xtractr index also contains the time and packet spans of each flow, which is critical to the visualization below.

Visualizing Application Flows

We are in the middle of a node.js, CouchDB project that uses the RESTful xtractr API to stream packets out of the index as nice little JSON documents. With some jQuery awesomeness, the following short video is a pure-HTML/CSS animation of streaming packets from xtractr and visualizing the dynamic nature of networks.

To paraphrase Cypher from the Matrix:

The network has too much data. So the only way to see what’s going on is through this console. Pretty soon you see a file download, people tweeting, IM’ing each other, playing games on Facebook, etc.

You can check out the API using the Ruby bindings available on googlecode. Let us know what you think!

Categories: Forensics, jQuery, Video, Visualization, xtractr
1,200 Comments
Bookmark and Share