Testing Various Configurations of Rails, Merb, Swiftiply, and Nginx
If you're developing Web applications in Ruby and are concerned about system scalability, you may find this post useful. I've been exploring different ways to optimize front end application performance before setting my sights on the back end. Along the way, I've compared the performance of Rails and Merb using various architectures for handling HTTP proxying, session management, and logging. The results are provided in the second half of this article.
But first, an introduction to the components involved...
Merb is the Word
Merb is an alternative to the Ruby on Rails MVC framework. It also includes an optimized Mongrel handler. I like Merb's simplicity. The trade off is that you will need to forgo some of the out-of-the-box features you may depend on in Rails. One suggestion is to leverage Rails for quick and dirty proof of concepts, then switch over to Merb for building your production app.Fast, Lightweight Nginx
As you probably know from Rails best practices, to handle real world Web usage scenarios, you'll need to run a cluster of Mongrel processes on each of your application servers. Apache is a solid, fast HTTP server that can be used to proxy incoming requests to your Mongrel clusters. However, when you've got multiple sites/projects hosted on one server, memory usage is a big deal. Even if you're running a single site, too many concurrent connections can evaporate memory quickly. Thus, I prefer using Nginx as an HTTP proxy, since its memory footprint is very small, it can handle a pounding or two, and serves up static content super fast.Swiftiply Raises the Bar
Good things come in pairs. Soon after Merb, I discovered Swiftiply - another HTTP proxy. What's interesting is that it supports two types of configurations. One transforms each of your Mongrel instances into a Swiftiply client, enabling a persistent connection to the proxy. Under this config, the Swiftiply server can handle HTTP directly and quite well in fact (see the Swiftcore benchmarks). In my tests, this configuration is referred to as "Swiftiplied Mongrels."The second mode of Swiftiply makes use of EventMachine, a nifty event processing lib, instead of threads. Under this implementation, you don't run the Swiftiply server --- all you do is supplement Mongrel with the "Evented" version (via config setting) and run your cluster like usual. Hence, this configuration is called "Evented Mongrels."
For both configurations, I was curious about adding Nginx into the mix. I've posted configuration details for running Nginx with Swiftiply in a separate post.
Testing Methodology
My goal was to test clustering configurations one would expect to see in a production application. I specifically looked at how concurrency affected the application running Rails vs. Merb, various session management options, and logging. The take home message in these tests is that the numbers are not meant to be absolutes; they provide insights into the relative differences between various Ruby-centric architectures.The application was a simple one -- it consisted of a view returning the current time. The only dependency on the back end was present during session management testing, as you'll see below. My goal in the future is to conduct full end-to-end tests comparing things like data abstraction layers but for now, let's keep the focus on the framework and app servers.
Some additional testing notes:
- Server specs: AMD Athlon 64 Dual Core 3800 1GB RAM running Fedora 6
- Testing client: Apache Bench 2.0.40-dev running on same machine hosting the test app
- All tests used 127.0.0.1 as the base URL
- Mongrels were run in "production" mode
- Mongrel logging was set to "info" unless indicated otherwise
- Each test run used 4 Mongrel processes
- Nginx used 4 worker processes; access and error logging were enabled
- For session testing, data was cleared out between runs
- Each test consisted of 5,000 requests of varying concurrency, as described below
And Now the Results...
Scenario #1: Sessions Disabled
You might have an application that acts as a Web service, returns a feed that changes infrequently, or does not have any user-specific functionality. Do you really need to support sessions in this case? Probably not. So, this first test compared Rails and Merb with session management disabled.| Framework | App Server | HTTP Server | Concurrency | Mean Requests/sec |
| Ruby on Rails | Mongrel | Nginx | 10 | 432.08 |
| Merb | Mongrel | Nginx | 10 | 471.36 |
| Ruby on Rails | Mongrel | Nginx | 100 | 388.40 |
| Merb | Mongrel | Nginx | 100 | 391.47 |
| Ruby on Rails | Swiftiplied Mongrel | Nginx | 10 | 440.88 |
| Merb | Swiftiplied Mongrel | Nginx | 10 | 510 |
| Ruby on Rails | Swiftiplied Mongrel | Nginx | 100 | 460.67 |
| Merb | Swiftiplied Mongrel | Nginx | 100 | 524.62 |
| Ruby on Rails | Evented Mongrel | Nginx | 10 | 528.17 |
| Merb | Evented Mongrel | Nginx | 10 | 594.84 |
| Ruby on Rails | Evented Mongrel | Nginx | 100 | 520.33 |
| Merb | Evented Mongrel | Nginx | 100 | 627.36 |
As you can see, Merb edges out Rails in all of the tests. But notice how Swiftiply helps both frameworks handle increasing concurrency better than a standard Mongrel cluster.
Scenario #2: Sessions Stored in the Database
When running a production cluster incorporating failover and load balancing strategies, you'll find yourself with multiple front ends. Rather than persisting sessions on each server, you'll want to take a more stateless approach and push this responsibility into the back end. Both Rails and Merb support ActiveRecord-based sessions; user data is stored in a 'sessions' database table.| Framework | App Server | HTTP Server | Concurrency | Mean Requests/sec |
| Ruby on Rails | Mongrel | Nginx | 10 | 227.79 |
| Merb | Mongrel | Nginx | 10 | 243.89 |
| Ruby on Rails | Mongrel | Nginx | 100 | 222.88 |
| Merb | Mongrel | Nginx | 100 | 224 |
| Ruby on Rails | Swiftiplied Mongrel | Nginx | 10 | 239.49 |
| Merb | Swiftiplied Mongrel | Nginx | 10 | 278.57 |
| Ruby on Rails | Swiftiplied Mongrel | Nginx | 100 | 235.76 |
| Merb | Swiftiplied Mongrel | Nginx | 100 | 280.30 |
| Ruby on Rails | Evented Mongrel | Nginx | 10 | 248.29 |
| Merb | Evented Mongrel | Nginx | 10 | 292.91 |
| Ruby on Rails | Evented Mongrel | Nginx | 100 | 260.57 |
| Merb | Evented Mongrel | Nginx | 100 | 317.45 |
Not surprisingly, performance dropped but Swiftiply helped with concurrency.
Scenario #3: Sessions Stored in Memcached
Memcached is a popular distributed memory caching solution. Companies like Facebook are rumored to be using several hundred Memcached servers to prevent back end bottlenecks. Memcached can also be used to persist session data, both Rails and Merb support it as a configuration option, assuming you first install the Memcached client gem. The philosophy is the same as in the previous set of tests... we want stateless application servers and hopefully with better performance than ActiveRecord-dependent sessions.| Framework | App Server | HTTP Server | Concurrency | Mean Requests/sec |
| Ruby on Rails | Mongrel | Nginx | 10 | 367.77 |
| Merb | Mongrel | Nginx | 10 | 409.50 |
| Ruby on Rails | Mongrel | Nginx | 100 | 346.30 |
| Merb | Mongrel | Nginx | 100 | 357.60 |
| Ruby on Rails | Swiftiplied Mongrel | Nginx | 10 | 371.61 |
| Merb | Swiftiplied Mongrel | Nginx | 10 | 483.14 |
| Ruby on Rails | Swiftiplied Mongrel | Nginx | 100 | 371.80 |
| Merb | Swiftiplied Mongrel | Nginx | 100 | 482.64 |
| Ruby on Rails | Evented Mongrel | Nginx | 10 | 419.86 |
| Merb | Evented Mongrel | Nginx | 10 | 560.39 |
| Ruby on Rails | Evented Mongrel | Nginx | 100 | 438.33 |
| Merb | Evented Mongrel | Nginx | 100 | 588.92 |
So clearly Memcached is a nice option when you need sessions enabled.
Logging vs. Performance
Logging can have a noticeable effect on application performance. If your application is writing vast amounts of data to the log frequently, I/O bottlenecks will occur. To see the opposite effect, I changed Merb's logging mode to only write "Fatal" errors --- the throughput increased to 760.53 requests per second. Of course, in a real world production app, logs can provide valuable insights into problems, so I do not advocate using anything less than the "info" level. Consider offloading logging functionality into background processes which write data to external data sources. I'll explore some of these options in detail in the future.What about the Overhead of Nginx?
Some of you may be wondering how things would look if we ran a cluster of Swiftiplied Mongrels solely using the Swiftiply server and eliminated Nginx. Well, it turns out the overhead is negligible --- in my tests I observed a net savings of 5-7 requests per second on the average.Conclusion
Using Evented Mongrels with Merb gives you the best bang for the buck overall when high concurrency is expected. If you're using sessions in your application, then give Memcached a look.As a parting thought, be sure you understand your application's usage patterns and not over-engineer your solution. In most cases, running Rails with a standard Mongrel cluster may be just fine for you. However, I hope I've helped inform you about other viable options.
Posted in: eventmachine, frameworks, merb, mongrel, nginx, performance, ruby on rails, scalability, swiftiply on Monday, August 20, 2007 at at 11:15 PM
Hi. Nice writeup. There are some performance and capability improvements coming down the pike for Swiftiply. Wanted to release last week, but didn't get it done, so looking at this week.
You should also give it a try without nginx in front of it. I'd be curious to see what numbers you get in that configuration compared to nginx + evented mongrels.
Also, on the topic of logging, you might take a look at Analogger. In my benchmarks, it's about twice as fast as using the standard Ruby Logger class. It is due for a little love and a 0.6.0 release sometime really soon now, too.
hi, great stuff.
Quick question: how does each of
these configurations perform
when near full capacity. What
if you added more mongrels?
The scenario I'm trying to understand is: is there a hard limit for the system that basically causes requests
to start getting dropped or is there way to detect that you are near
capacity and turn up another
mongrel in the meantime?
That is something that I am working on for the 0.7.0 Swiftiply release.
Swiftiply itself will queue up as many requests as it has file descriptors for. If you are running on a platform that supports epoll (Linux 2.6.x), it will use epoll with a default of 4096 descriptors, but you can configure that to be as many as you want. On other platforms, select() is used, which limits it to 1024 descriptors.
Swiftiply knows how many requests are queued up, waiting for a server to handle them, though, and so what I am working on are capabilities for dynamic cluster management. Basically, Swiftiply recognizes when requests are coming in faster than the backends are handling them, and it asks the cluster manager software to increase the number of backends. Alternatively, if the backends are doing nothing at all, it can ask the cluster manager to reduce the number.
Right now, with Swiftiply, if you are running on a platform with epoll support (Linux 2.6.x), you can configure it to support LOTS of concurrent connections (it'll default to 4096). Connections which can not be handled by a backend right away will be queued, with a default timeout on those queued connections of 3 seconds. That is configurable, though. Connections which are dropped are given a 503 Server Unavailable response.
One thing it appears you didn't consider is that when using a database adapter which is thread safe (such as sequel) merb can run without any mutexes at all. This greatly increases the capacity of a single merb instance vs a single rails instance. I would be curious to see you benchmark with merb/sequel. I think many of the people looking to use merb may drop active record for the same reason they dropped rails. They want flat out speed and don't care if they have to do a little more work to get it.
See http://blog.inquirylabs.com/2007/08/02/magnificent-merb-and-sequel/
Being able to run multithreaded doesn't do a lot for the overall throughput.
A given process can only do so much work. If, in the course of doing that work, there is an external latency, such as waiting on some IO, that can be done in a non-blocking way, then it's possible to get more work out of that process per unit time by using threads.
The work involved in handling web browser requests and rendering responses back to them, though, doesn't tend to fall into this category. It tends to be CPU intensive, and when one is waiting for something external, like a database query, one is usually waiting inside the body of the low level extension based database driver, so thread context switching doesn't happen there, anyway.
The most efficient way of dealing with these sorts of things, from a throughput perspective, is to avoid the overhead of threads completely and just pound them through, one at a time, with 100% of the process resources working on that single request at a time.
When you are talking about very small scale deployments (i.e. one or two backend processes), there is a potential user experience drawback that may come up, depending on the application, though.
If the application has some actions which are fast, and some which are very slow, and one is running a single multithreaded backend to handle the requests, the fast requests can still come in, get handled, and get their responses out to the user even if there is a slow request being slogged through at the same time.
In the single request pipeline, though, those fast requests are going to queue up behind the slow one, waiting for it to finish. In absolute close time, that single request pipeline will get through the typical web request work faster, but the user experience, for anyone who was waiting on what should have been a fast action, will be worse.
Hi Phil,
Nice work. I just did a simple flex chart based on these results. Have a look at http://nlakkakula.wordpress.com/2007/09/02/performance-comparison-for-rails-and-merb/
Thanks for your comments and additional info. I've been swamped with a few other projects but plan on doing another round of testing with some other configurations in the future.
Once any significant real-world work is happening the burden of sessions becomes proportionately tiny. I reckon that applies generally to these figures. I wouldn't make a decision based on this. Much better to mock up a real-world styled controller and see what happens.