Testing an Application, Infrastructure and a Service

July 14, 2010 by kowsik

This one’s been in the works for a while. I keep comparing application testing to infrastructure testing and then pondering where the major revolution is happening right now and I can’t help but talk about it. Since I don’t do well with numbers and tables (*yawn*), I thought I’ll doodle a few graphs to visualize the differences. Besides, paraphrasing Calvin, a blog is authoritative and written by a professional when it has charts! :-)

Taxonomy

Before we dive straight down into the details, let’s talk about what the difference is between an application, infrastructure and a service.

taxonomy.png

Okay, that one’s easy. The blue pill is all about the real world, people, dogs, pets and web applications. The red pill though is those pesky machines that seem to talk to each other. So let’s get this straight: In this blog, apps are about what the user interacts with, infrastructure is the thing that pushes bits around and finally what’s this new fangled service thing? We’ll get to it at the end.

Who’s who

Well, we can’t compare and contrast until we bring on the players. No surprises here, on the applications side we have HP, IBM, CA, etc and on the infrastructure side we have IXIA, Spirent, etc.

whos-who.png

And you know whodat‘s on the top right corner!

Experience

So the purpose of testing these systems is for what? Turns out the application testing tools are worried about the user experience, while those on the infrastructure are worried about the network. Interesting… Aren’t those converging in some ways? Give me a single firewall these days that doesn’t do App-ID, DPI, IPS, URL Filtering, Anti-virus, Mail-filtering and Anti-something-or-the-other and I’ll be shocked! Anyways, the obligatory chart…

experience.png

Complexity

Why are application and infrastructure testing tools so different? There’s a reason: it’s transactional vs. protocol complexity. Repeat after me. Transactional. vs. Protocol. Complexity. You see, HP, IBM and CA don’t have to deal with ASN.1, PER, length’s, checksum’s, crc’s, offsets, p-strings, c-strings, etc, etc. Most of it is HTTP with some Oracle and Tibco stuff thrown in. However, their notion of a transaction is very long. Imagine executing an entire business process culminating in billing! The IXIA’s and Spirent’s though, they are masters of the OSI layers. They got the RFC’s, detailed specifications of how things ought to work with precision and they have teams that know the difference between bit-wise AND and bit-wise OR.

complexity.png

Metrics

Testing tools help you measure something or the other. Otherwise, they are just a tool and nobody has a budget line item to buy them. Turns out application and infrastructure testing look at very different, yet co-existing aspects of the same thing.

metrics.png

If you talked to a database guy then all s/he cares about is TPS (transactions/sec). When was the last time you measured the SQL performance with packets/sec? Now you talk to the routing guy and s/he goes OMG, what the $()*# is TPS? I need 64-byte packets, low latency, high throughput at a gazillion packets/sec with no jitter ‘cos my ASIC just kicks $%^. Okay then.

Rate of Change

This is an important dimension. Applications just move at the speed of light. There’s new stuff on the iPhone and Android, like, daily. 10′s of 1000′s of them. And JSONP didn’t happen through a committee. Some dude hacked it up, people like it, updated their servers and now we have cross-domain Ajax requests. Guess when the last TCP RFC was published? B.e.f.o.r.e. m.y. t.i.m.e. Fine, I’m lying, but in any case:

rate-of-change.png

What this really means though is that your test tool’s gotta test stuff even before there’s a specification or a standard, ‘cos people are already deploying it anyways and getting stuff to work.

Okay, last one: The Trend

Where do you think we are trending to? We got cloud deployments, virtualization, exploding number of mobile applications, social networking, social gaming, application identification, deep packet inpection, voice-over-ip, iptv (aka as netflix instant watch), and a large number of next generation services like long-term-evolution (LTE or 4G), smart-grid, etc aiming to please you the user.

the-trend.png

So what we really have are services that blend in all the transactional complexities of applications with the protocol complexities of the infrastructure, that change as fast, care about the user experience and fail whales and yet the test tool vendors are absolutely stuck doing nothing! Well, they are taking their age old stuff, moving it to the cloud and calling it ready, but really? The infrastructure guys are so obsessed with chassis, port-density and protocols, they don’t even know what’s hitting them! *sigh*

Just read that last paragraph again. Man, that was harsh. But still, so, anyways, what was I saying? Did I tell you Mu Studio is the coolest test tool, like, you ever saw, for testing next generation services? Really. You have to check it out.

Categories: Rants, Studio, Testing
1,640 Comments
Bookmark and Share