At blitz.io, for a while there, we were only relying on CouchDB clusters as the primary NoSQL database with some in-memory caching. As we grow (rapidly) and scale out, there are aspects of what we collect and store that are transient and real-time. While CouchDB is awesome for the map/reduce, replication and incremental view indexes, the real-time queues (emails, counters, stats, etc) natural lend themselves to, yup, redis. We are in the process of rolling out geo-located redis instances as part of our global infrastructure.
Archive for CouchDB
Help CouchDB break the C10K barrier
Over the weekend, I was experimenting with CouchDB to see if it can pass the C10K barrier. Some of the performance optimizations I made along the way are really OS-level optimizations that affect MochiWeb (erlang web server) and fairly well documented in many blogs. This one by @metabrew in particular is a pretty good read, since it focuses on Erlang and MochiWeb. While I am a performance junkie, I am not an Erlang hacker. So this is a call for help to the CouchDB hackers for recommendations on scaling out CouchDB.
blitz.io – Path-finding with CouchDB
blitz.io went down for a short duration yesterday morning. It was an interesting day uncovering and identifying issues we hadn’t encountered before with multi-region CouchDB clusters that are doing multi-master continuous replication. In a lot of ways, we are path-finding and pushing CouchDB to its limits given that we are a write-heavy app. In the process, we are making up our own best practices and working around issues. Some of these issues are already addressed in trunk, but I wanted to document what we went through today and what we can do about this. Any ways, if you are running a large CouchDB cluster in production, would love to hear from you.
blitz.io: How we use Heroku, AWS and CouchDB
I put together this prezi a few weeks ago as a quick way to describe the internals of blitz.io. This blog is an expansion of this to go into further details on the internals. We launched blitz.io a few months ago to really bring load and performance testing to developers, as part of the continuous integration. Last week we released multiple API clients in various languages to make this possible. We realize that most cloud-based load testing tools are heavy and are geared towards experts and cost significant time and $$$ to do performance testing. With the rise of PaaS, it’s imperative that this type of testing is easy, affordable to the developers and really part of dev and test, not a one off expensive event that happens once a year.
blitz.io wins the best CouchDB App at CouchConf
So I head up to the High Sierras for a week switched off from the grid and on the way back stop at a Chinese restaurant for lunch. My fortune cookie says “You will be pleasantly surprised” or some such thing. I kid you not. Turns out blitz.io has won the best CouchDB App at CouchConf and we are super excited about this. While I don’t know exactly the criteria used by the panel, I sure do know that blitz.io pushes CouchDB to the limits in many ways. Beyond just the map/reduce, we use lots of other cool things about CouchDB in production, making blitz.io the first multi-tenant load testing solution targeted at app developers.
The day DevOps became NoOps
The Internet haz gone down, at least parts of it. Last night, the Virginia EC2 region started experiencing all sorts of issues with their EBS storage (some suspect it’s the bandwidth) and we are still feeling the impact. For blitz.io (which is down, BTW), we use Heroku that runs off of the Virginia region. When we first architected blitz.io, we were afraid of regional failures and the database bottlenecks and ended up deploying an entire CouchDB cluster across multiple regions with master to master replication.
Announcing pcapr.Local
Since the launch of pcapr.net a few years ago, the packet-geek community has completely embraced it, extended it and now we are at over 60+ million packets on the cloud, serving enterprise IT folks, operators, government agencies and security/packet geeks. To those that seek specific packet samples, pcapr.net serves as a major reference with samples of over 420+ protocols, full-text search and the automatic indexing and organization of pcaps. But…
blitz.io: Deploying CouchDB in the cloud, literally
Ever since we launched blitz.io a couple of weeks back, we’ve been thinking. Yeah, we are using CouchDB with multi-master replication to handle geo-latencies and we got five regions around the world with AWS that we can generate serious load from. Frankly though, we are not happy with our deployment. We really need to be right next to the data centers where these apps are deployed so we can instantly generate massive loads with very low latency.
blitz.io: DevOps, NoOps and Cloud Automation
If you haven’t read the interview with @DEVOPS_BORAT on devops.com, check it out. ROFL-funny but within those hilarious interview is lots of wisdom about devops. For one, it’s this whole new movement on the cloud where a few developers are both writing the code, provisioning systems, managing and monitoring them and automating the hell out every aspect of the entire system. blitz.io is no different. In under two weeks post-launch, blitz-users have generated 10,000,000 hits against their servers, added varnish and memcached to their app, resolved various lock contentions, and continue to tweak and optimize the code, all by themselves! We do think load and performance testing should be this fun and easy!
blitz.io: Benchmarking view performance in CouchDB
If you are running CouchDB on an EC2 node with an EBS volume (or otherwise), it’s kinda hard to predict how the view performance will be affected by disk seeks and other criteria. Add complex reduce functions to this mix and you are flying blind. Just in my last blog, we talked about adding parameterization to blitz.io so you can easily test location-aware iPhone and Android apps. What if we could use CouchDB itself to generate fully parameterized tests so you can blitz it?
