We launched blitz.io a couple of days ago, fully powered by CouchDB. In the last couple of days, we’ve generated over 1,000,000 hits (through our load and performance testing) against various servers and transfered over 2GBytes of data in and out EC2. Unlike massive terrabyte read-only CouchDB clusters, ours is inherently write-heavy. When each user either through the UI or through the API runs a sprint or a rush, we have multiple writes to CouchDB, constantly.
Archive for CouchDB
Announcing blitz.io – and we are hiring!
We just launched our public beta of blitz.io. We wanted to bring the excitement and simplicity of git push for cloud deployments to load and performance testing. As a matter of fact, we think performance testing ought to be a social sport! Most load testing solutions on the cloud today are built heavy (remnants of the J2EE world), require major investment in time and resources, not to mention you have to talk to an expert to accomplish anything meaningful. By adapting common dev tools and dramatically simplifying this, we hope to make cloud performance testing incredibly easy.
Notes and tips on optimizing CouchDB performance
We’ve been using CouchDB for a couple of years now, starting with pcapr. Couch was still at version 0.8 when we first started using it since then it has come a long way. And so have we. We are actively using Couch both in the cloud (pcapr and Test Cloud) as well in our product. With the last release of Mu Studio, we can now daisy chain any number of appliances to generate truly elastic scale. All the engines are coordinated through the master which is running Couch and we use it to map/reduce massive amounts of statistics collected by the scale engines.
Having said that, the following are tips and various notes on squeezing out all the performance out of Couch. This is not just configuration settings on Couch, but more like an end to end set of tweaks. Do leave a comment if you have more suggestions.
Categories: CouchDB
From Premise to Packet
My name is Rob Cameron. I am the product line engineer manager at Juniper Networks for the data center SRX. As a product line engineer we are experts in our particular technologies. We use this to make the best possible products and empower the field with the tools they need to best help customers. I have been working in the industry for over twelve years and have have authored five books about networking and security. My latest book is Junos Security from O’Reilly. I am very happy to be a guest blogger on Mu Dynamics Research Labs blog. Full Post »
Optimizing node.js/CouchDB apps with Studio Scale Test
As most of you probably know, the Slashdot Effect is caused by someone posting a link to Slashdot which triggers massive amounts of inbound requests that melts the web server. These days it could be Twitter, Reddit, HN, etc. To quote the Wikipedia:
Few definitive numbers exist regarding the precise magnitude of the Slashdot effect, but estimates put the peak of the mass influx of page requests at anywhere from several hundred to several thousand hits per minute.
This short video shows you how to create the Slashdot Effect using Studio Scale.
Using CouchDB group_level for hierarchical data
CouchDB supports something called group_level in the view queries. On pcapr, we never really had the need to use this feature though we have over 52 different views. But in a recent internal project, we had the need to display folders in the application that can be expanded and collapsed. Each document in CouchDB represents a file of sorts and contains the relative path name. One of the views in the app is a classic folder view that can be expanded recursively. Obviously, from a scaling perspective, we don’t want to load all this data up front and that’s exactly where the group_level comes in. This was my first time playing with this capability and I have to say, once you get grok this, it’s totally cool.
Categories: CouchDB
CouchDB, DNS and Scaling the Cloud
Just got back from Interop where I was part of a panel that talked about cloud computing. We discussed a lots of interesting topics like migration, scaling, hybrid clouds and what not. NoSQL was definitely a discussion point since I personally believe you can’t talk about cloud without also talking about NoSQL.
The scaling part though got me thinking. The current approach for scaling any cloud app is to use your IaaS provider to just add more compute power and deal with it. I tend to think a little differently from this. xtractr on pcapr for example, uses a hybrid cloud model. You download a single binary that you use for indexing large packet captures. When you now want to search, extract, report on this, the application is delivered to your browser which then uses JSONP (until HTML5 is truly prevalent with cross-domain Ajax requests) to communicate to your instance of the xtractr. What this means is when you are busy crunching packets, the server load on pcapr is zero! Which implies infinite scaling, ‘cos the load is truly distributed across all of our users.
Why NoSQL is bad for startups
We launched pcapr over a year ago now with just a few of us working part time to build and manage the site. pcapr is powered by CouchDB, a NoSQL database written in Erlang with JavaScript as the primary query language. Frankly, this has been a disaster. We are planning on rebuilding the site with Java, Hibernate and MySQL for a number of reasons.
Using Map/Reduce for Network Forensics and Troubleshooting
We launched xtractr earlier this week for network forensics, troubleshooting and handling support escalations involving large packet captures. Just so you know xtractr is a 4-tier app (more on that below) that combines the best of Web 2.0 with looking at packets in new light. Looking beyond the “unleash the power of packets” message, I wanted to write about what’s under the hood a little bit and how we are using CouchDB-style of Map/Reduce for uncovering all sorts of information inside large packet captures.
Multi-dimensional data visualization
Way back in grad school, I was working on a project involving Auralization. The key idea was that your ear can process multi-dimensional data (pitch, volume, instruments, silence, tempo, etc) way better than your eyes can (try closing your eyes and listening to a Bach Fugue). So back then, we tried to take these types of data (stocks, sales reports, expenses, etc) and created MIDI files out of it to understand trends. Ever since I saw the Hans Rosling’s TED Talk I’ve wondered the applicability of this type of visualization on something other than economics.
