Webficient Blog - Home

Thin Web Server Benchmarks using Ruby on Rails

2008-06-30T23:29:00Z

In a previous article, we looked at the performance and memory usage of a Ruby on Rails application running on Passenger Phusion (mod_rails). In this post, we’ll compare the same app running on the Thin Web server behind an Nginx cluster.

Thin Basics

Thin uses the Mongrel parser and Event Machine I/O library. In addition, it uses the Rack interface, allowing you to run other frameworks besides Ruby on Rails.

In a proxying scenario, where Nginx is the “front-end” for the Thin application server, you can leverage Unix domain sockets, which can boost performance and memory usage. Not all HTTP load balancers support it though.

How We Tested

To recap our testing configuration, we’re running a 512MB VPS slice (courtesy of Slicehost.com), and using Apache Bench to load test the login page of our Ruby on Rails blogging platform. This page is not cached but there isn’t a database call either, so our performance numbers don’t include any backend interaction. For other details, refer to the “Benchmark Overview” in our previous post.

Our first test used 4 Unix domain sockets, which are configured as follows, in nginx.conf:

upstream webficient {
  server unix:/tmp/thin.0.sock;
  server unix:/tmp/thin.1.sock;
  server unix:/tmp/thin.2.sock;
  server unix:/tmp/thin.3.sock;
}

Then, Thin was started up using

thin start -s4 --socket /tmp/thin.sock -e production

Results

Memory usage under load using Thin/Nginx appeared better than mod_rails (with Ruby Enterprise Edition).

12	total used free shared buffers cachedMem: 524460 458272 66188 0 7060 82792

Requests per second, on the other hand, were a bit slower than mod_rails (246 RPS vs. 292 RPS with mod_rails) but significantly faster than standard Mongrel.

Dynamic Content Load Test – Thin/Nginx

Concurrency Level:      100
Time taken for tests:   40.599507 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      30210000 bytes
HTML transferred:       25690000 bytes
Requests per second:    246.31 [#/sec] (mean)
Time per request:       405.995 [ms] (mean)
Time per request:       4.060 [ms] (mean, across all concurrent requests)
Transfer rate:          726.63 [Kbytes/sec] received

Seeing that we had a surplus of available memory during load testing, we tweaked the number of Unix domain sockets to evaluate the effect on overall performance. However, going from 4 to 5 sockets did not improve performance significantly. Dropping down to 3 sockets decreased performance. Therefore, our recommendation for a 512MB VPS slice is 4 sockets. In another test, we doubled Nginx’s number of worker_processes from 3 to 6, however, this did not significantly improve Thin’s numbers either.

Conclusion

While slightly slower than mod_rails, Thin significantly outperforms a vanilla Mongrel Cluster/Nginx configuration. Thus, Thin is a viable solution to those who prefer Nginx to Apache. Recall that previous benchmarks show Nginx is the clear leader in serving up static content. So if your site uses caching extensively, you may want to consider this setup. Otherwise, Phusion Passenger will give you the best bang for the buck.

Another consideration is stability. Thin has been out for quite some time and has had 7 or so releases. Mod_rails is relatively new and not without issues, as seen in its Google group and blog posts. We’ll expect things to improve quickly in this area.

Using Ruby Enterprise Edition with Nginx/Mongrel

2008-06-30T23:00:00Z

In a continuing series which benchmarks Phusion Passenger (mod_rails) against other popular Ruby Web serving configurations, today we’re looking at how a vanilla Mongrel cluster backend/Nginx frontend perform using the recently released Ruby Enterprise Edition.

For testing methodology and machine specs, please refer to the first article in the series.

No Impact on Memory Usage

As already stated in the Phusion Passenger FAQs, memory usage will not improve simply by using Ruby Enterprise Edition with Mongrel. This is due to the fact that Mongrel cannot leverage Ruby EE’s memory optimization routines without patching. It’ll be interesting to see this hurdle addressed by the community.

Performance about the Same

A Mongrel cluster powered by Ruby Enterprise Edition performed on par with a standard Ruby 1.8.6 virtual machine running on a 512MB VPS slice powered by Mongrel and Nginx.

Concurrency Level:      100
Time taken for tests:   62.589211 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      30370000 bytes
HTML transferred:       25690000 bytes
Requests per second:    159.77 [#/sec] (mean)
Time per request:       625.892 [ms] (mean)
Time per request:       6.259 [ms] (mean, across all concurrent requests)
Transfer rate:          473.85 [Kbytes/sec] received

Conclusion

To realize the memory and performance numbers seen with previous benchmarks, your best bet is to use Ruby Enterprise Edition with Phusion Passenger/mod_rails.

Phusion Passenger Optimizations

2008-06-26T00:54:00Z

In a previous article, we installed and tested a baseline configuration of Phusion Passenger, aka mod_rails. In this post, we’ll be comparing memory usage and performance by varying Apache configuration settings in httpd.conf.

Realize that these numbers aren’t absolutes and will vary on your server, depending on its architecture, CPU, and RAM. Our VPS slice has the following specs:

Linux Fedora 8, 64-bit
512MB RAM
Rails 2.0.2
Phusion Passenger 2.0.1
Enterprise Ruby 1.8.6 patchlevel 11

We used Apache Bench for testing:

1	# ab -n 10000 -c 100 http://server/app_path

PassengerMaxPoolSize

PassengerMaxPoolSize specifies the maximum number of Ruby on Rails application instances that may be simultaneously active. This setting has a lot of bearing on overall memory consumption and performance.

Memory Usage

PassengerMaxPoolSize    total       used       free     shared    buffers     cached
        1              524460     417040     107420          0      21384     120216
        2              524460     454740      69720          0      22560     127080
        3              524460     489692      34768          0      23160     131860
        4              524460     494548      29912          0      19860     110868
        5              524460     516772       7688          0      19100     103628

Performance

PassengerMaxPoolSize         Requests per second 
        1                    89.09 [#/sec] (mean)
        2                   170.28 [#/sec] (mean)
        3                   224.88 [#/sec] (mean)
        4                   274.94 [#/sec] (mean)
        5                   272.45 [#/sec] (mean)

You’ll notice that our sweet spot is when PassengerMaxPoolSize = 4, since we did not gain any more performance when set to 5, and less memory is used.

PassengerMaxInstancesPerApp

According to the docs, PassengerMaxInstancesPerApp is the maximum number of application instances that may be simultaneously active for a single application. The recommendation is to have the value be less than PassengerMaxPoolSize. Trying different values, ranging from 1 to 3, we observed memory consumption to be a bit less, however the number of requests per second also suffered. We found that setting the value to 0, or no limit, produced the best results performance-wise. If you’re experiencing heavy swapping on a smaller slice, on the other hand, capping this value at something between 0 and PassengerMaxPoolSize will help.

Things that May Affect Benchmarks

If you’re curious about seeing the impact of the above settings on your own configuration, be aware of the following.

Heavy Swapping

As physical memory is used up, your machine will start swapping virtual memory to disk. So be aware of any background processes already running. Disable any cron jobs, scheduled backups, etc.

Linux IP Tables

A software firewall like Linux IP Tables tracks each connection and temporarily persists a handle in a built-in database. It’s entirely possible to overwhelm the system, which will affect your numbers. Your Linux kernel log will show this message: “ip_conntrack: table full, dropping packet.” For an explanation of this issue and how to solve it, see this article.

Ruby Enterprise Edition and Passenger

2008-06-24T08:35:00Z

Recently, we evaluated Ruby Enterprise Edition and Apache mod_rails, known as Phusion Passenger, in a Virtual Private Server hosting environment. We compared performance and memory usage against our production instance, which runs this blog and uses an Nginx-powered Mongrel cluster.

[Editor’s note: new results have been published as of June 30, 2008.]

Preparation

First off, doing these kinds of evaluations is super easy with a VPS hosting service like Slicehost. We simply cloned a image of an existing production backup rather than building one from scratch. Then, we shut down Mongrel and Nginx.

Installation

Ruby Enterprise Edition

This was a breeze, following the installation instructions.

# cd /usr/local/src
# wget http://rubyforge.org/frs/download.php/38803/ruby-enterprise-1.8.6-20080624.tar.gz
# tar xzvf ruby-enterprise-1.8.6-20080624.tar.gz
# ./ruby-enterprise-1.8.6-20080624/installer

The installer will check for the GNU C++ compiler, and the Zlib and OpenSSL development headers, which are easy to install if you’re using a package manager like Yum on Fedora Linux.

The only hiccup we ran into was during the installation of the Enterprise Ruby-flavored version of the MySQL gem, getting the infamous “Failed to build gem native extension.” Fortunately, the rest of the installation process went through smoothly. Afterwards, we used this command to install the MySQL gem:

1	# /opt/ruby-enterprise-1.8.6-20080624/bin/ruby /opt/ruby-enterprise-1.8.6-20080624/bin/gem install mysql -- --with-mysql-include=/usr/include/mysql --with-mysql-lib=/usr/lib64/mysql

Important: The Ruby Enterprise Edition executable is installed in a separate location so that you end up with two versions of Ruby on your machine. This means when installing new gems, you should use Ruby Enterprise Edition version. You can make this the default version by updating your environment settings:

1	export PATH=/opt/{ruby enterprise edition folder}/bin/:$PATH

Another gotcha is that the Ruby Enterprise Edition installer will install Rails 2.1.0. If you want 2.0.2, you’ll need to install it with

1	gem install rails -v 2.0.2

Passenger (mod_rails)

Again, the installation process was simple per the docs.

First, the gem:

1	# gem install passenger

Then, running the following command kick offs the installlation wizard.

1	# passenger-install-apache2-module

Again, the installer checks for required dependencies and will advise you how to handle missing ones. In our case, we did not have Apache 2 and its development headers installed on our slice (since we were using Nginx).

The installation wizard will also tell you next steps for configuring Apache, which involves copying a few lines of settings. And to make Apache aware of your Rails app, it’s as simple as setting up a Virtual Host.

1234	<VirtualHost *:80> ServerName www.webficient.com DocumentRoot /u/apps/webficient/current/public</VirtualHost>

Don’t forget to restart Apache for the changes to take:

1	# service httpd restart

(You can also use this technique if you don’t want to restart the entire httpd process).

Benchmark Overview

Because our blogging platform caches content, we looked at both dynamic and static content performance to ensure the numbers weren’t misleading. Our dynamic test involved hitting the login page. Our static test requested an existing cached article.

We used Apache Bench to simulate load:

1	# ab -n 10000 -c 100 http://server/app_path

Both slices had the following aspects in common:

Linux Fedora 8, 64-bit
512MB RAM
Rails 2.0.2

The differences were type of application server and Ruby VM:

“Production” slice uses 3 Mongrels (v1.1.5), 3 Nginx (v0.6.31) worker processes (with 1,024 worker connections), and Ruby 1.8.6 patchlevel 114.
“Test” slice uses Passenger (mod_rails) and Enterprise Ruby 1.8.6 patchlevel 111. Based on the recommendations in the Passenger documentation, we settled with a PassengerMaxPoolSize of 4, given the size of our VPS.

Memory Usage

Since we’re always interested in squeezing out every last meg of RAM in our slice, we were curious to see how things fared under mod_rails and Ruby Enterprise Edition. Before running any benchmarks, it was clear that without the memory hungry Mongrels, our mod_rails test slice was quite a bit leaner.

Idle Comparison

123	total used free shared buffers cachedmongrel: 524460 382736 141724 0 29816 118924mod_rails: 524460 241684 282776 0 21856 121256

Be aware that Apache allocates very little memory until the first request comes in. You’ll expect to lose about 80 MB in free memory when that happens. So the above numbers will rarely be seen, unless you have a low traffic Web site.

As expected, memory usage surged during the load tests. With a PassengerMaxPoolSize equal to 4, mod_rails consumes more memory than the Mongrel/Nginx configuration. Setting it to 3 produced the opposite result. However, you’ll see this setting also affects requests per second.

Dynamic Content Test

123	total used free shared buffers cachedmongrel: 524460 481800 42660 0 27488 116984mod_rails: 524460 500948 23512 0 20496 115360

Static Content Test

123	total used free shared buffers cachedmongrel: 524460 439552 84908 0 29564 118908mod_rails: 524460 257604 266856 0 21544 121192

So under load, mod_rails can take it well. We are also thrilled that the mod_rails version of our blogging app is not leaking memory, unlike our Mongrel-powered production version.

Performance

What about speed? Results were in line with our expectations. Mod_rails does well serving up dynamic content but Nginx continues to dominate when static content is involved.

Dynamic Content – Mongrel/Nginx

Although the numbers shown here represent a PassengerMaxPoolSize set to 4, we also ran the same tests with a value of 3. Number of requests per second decreased while memory usage improved.

Concurrency Level:      100
Time taken for tests:   63.843779 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      30370000 bytes
HTML transferred:       25690000 bytes
Requests per second:    156.63 [#/sec] (mean)
Time per request:       638.438 [ms] (mean)
Time per request:       6.384 [ms] (mean, across all concurrent requests)
Transfer rate:          464.54 [Kbytes/sec] received

Dynamic Content – mod_rails/Ruby Enterprise Edition

Concurrency Level:      100
Time taken for tests:   34.190459 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      30900000 bytes
HTML transferred:       25690000 bytes
Requests per second:    292.48 [#/sec] (mean)
Time per request:       341.905 [ms] (mean)
Time per request:       3.419 [ms] (mean, across all concurrent requests)
Transfer rate:          882.56 [Kbytes/sec] received

Static Content – Mongrel/Nginx

Concurrency Level:      100
Time taken for tests:   1.901919 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      540173910 bytes
HTML transferred:       538042632 bytes
Requests per second:    5257.85 [#/sec] (mean)
Time per request:       19.019 [ms] (mean)
Time per request:       0.190 [ms] (mean, across all concurrent requests)
Transfer rate:          277358.28 [Kbytes/sec] received

Static Content – mod_rails/Ruby Enterprise Edition

Concurrency Level:      100
Time taken for tests:   2.550226 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      540430000 bytes
HTML transferred:       537720000 bytes
Requests per second:    3921.22 [#/sec] (mean)
Time per request:       25.502 [ms] (mean)
Time per request:       0.255 [ms] (mean, across all concurrent requests)
Transfer rate:          206947.55 [Kbytes/sec] received

We’re Not Done Yet

Although we like what we see so far, we’re going to run some additional tests and analysis, including:

We’ll post our findings in the near future.

Pre-Caching Rails Views During Application Deployment

2008-06-21T07:30:00Z

If you’re caching any pages in your Rails app as a performance enhancer, you may find this post useful. If your Web application gets a decent amount of traffic, then waiting for Rails to cache your Action View templates as HTML, post-deployment, is not ideal. While people are banging on your site, the CPU is spiking, things are queuing, and everything slows down.

We’ve cooked up a Capistrano recipe that leverages the Ruby Mechanize gem and crawls your application, starting at the site index or any page you specify. So instead of sitting there and clicking each link, learn about how you can use this Cap task to enhance your next project.

Features

Able to crawl local or remotely hosted application
Crawling limited to application pages, does not follow external links
Ignores links pointing to non-HTML content
Traps HTTP errors and moves on to next item

Installation

First, be sure you have the Mechanize gem installed:

sudo gem install mechanize

Next, inside your Rails app, create a new file at ../lib/tasks/crawler.rb:

require 'mechanize'

class Crawler
  
  EXTENSIONS_IGNORED = %w[.csv .doc .docx .gif .jpg .jpeg .js .mp3 
          .mp4 .mpg .mpeg .pdf .png .ppt .rss .swf .txt .xls .xlsx .xml]
    
  def initialize(starting_url, history_size = 1000)    
    @agent = WWW::Mechanize.new
    @agent.history.max_size = history_size
    @starting_url = starting_url
    extract_and_call_urls(starting_url)
  end

  def extract_and_call_urls(url)    
    #catch any non-html doc types up front and exit    
    return if EXTENSIONS_IGNORED.detect{ |ext| url =~ /#{ext}$/ } != nil
            
    #get page
    puts "url: #{url}"
    begin
      page = @agent.get(url)
    rescue => exception
      puts exception.message
      return
    end
    
    #for any content types we may have missed above, 
    #exit if content type is not html
    return if page.content_type.index('text/html') == nil 
    
    #get links found on page
    links = page.links

    #for each link, call the url if not in history
    links.each{ |link| extract_and_call_urls(link.href) unless 
      @agent.visited?(link.href) || invalid_url?(link.href) }
  end
  
  private
  
  def invalid_url?(url)
    (url.index('http') != nil || 
    url.index('ftp') != nil || 
    url.index('mailto') != nil) &&
    url.index('http://localhost') == nil &&
    url.index('http://127.0.0.1') == nil
  end
  
end

Now, you need to make the above functionality available as a Capistrano task. Add this line to the top of config/deploy.rb:

require '/lib/tasks/crawler'

And at the bottom of deploy.rb, add this task:

desc "Crawl pages using the Mechanize gem. Optionally set URL variable 
      as the starting point."
task :crawl_pages do
  start_url = ENV["URL"] || "http://localhost:3000"
  Crawler.new(start_url)
end

Usage

As with any other Capistrano task, you’ll need to use your command line shell within your application’s directory. Check out Webistrano if you prefer running Capistrano tasks from your browser.

There are several ways you can apply the crawl_pages tasks. You can run the command on your development machine, add the files to your source code repository, and they’ll be automatically included in your next deployment to production. This approach mostly makes sense for pages with static content.

cap crawl_pages

If, however, your content is dynamic (perhaps generated by a content management system), then you’ll need to run the task against your live site. The way the task is written allows you to run the command from your local machine by specifying a remote URL. However, the task limits itself to crawling only pages associated with the site.

cap crawl_pages URL=http://mysiteurl.com

Automatic Crawling

If you’d like the crawler task to fire off automatically, just add the following line to deploy.rb:

12	after :deploy, :crawl_pagesafter "deploy:migrations", :crawl_pages

Other Uses

Besides pre-caching pages, you can also use the crawler task to check for broken links and exceptions. Any HTTP exception will be trapped and displayed in the command line output. For exceptions, you can correlate the problem in the server log, or if you have the slick Exception Notifier plugin installed, you’ll receive an email with the juicy details.

Enjoy!

Keeping Domain Logic Out of Views

2008-06-20T08:55:00Z

A typical Rails application includes domain logic and presentation logic. Domain logic – often called “business logic” – includes algorithms, calculations, rules, etc. applicable to your models.

class Employee < ActiveRecord::Base
  def recently_hired?
    Time.now.months_ago(1) < self.hire_date
  end
end

Presentation logic, on the other hand, affects the rendering of elements in the user interface.

(/employees/_employee.html.erb):

<% div_for employee do %>
  <% if employee.recently_hired? %>
  ...employee details...
  <% end %>
<% end %>

The problem is when domain logic leaks into Views and begins to take on the responsibility of presentation logic.

<% div_for employee do %>
  <% if Time.now.months_ago(1) < employee.hire_date   %>
  ...employee details...
  <% end %>
<% end %>

The above rule may change, perhaps Human Resources decides “recently hired” includes anyone hired this past quarter. So in this case, a change in domain logic would require you to change your View code. Further, if you needed to evaluate this condition in other Views, you’d have unnecessary code duplication that now needs to be maintained in more than one spot.

Looking back at the second code snippet, you may be thinking, doesn’t the following smell like domain logic?

123	<% if employee.recently_hired? %>...employee details...<% end %>

And what happens when requirements change, and now we have to also consider employment status…

<% div_for employee do %>
  <% if employee.recently_hired? && employee.full_time? %>
  ...employee details...
  <% end %>
<% end %>

Again, it seems like changes to domain requirements would require us to maintain the View. This may not seem like a big deal when you’ve got a handful of templates, but in a larger application, where there may be a few developers actively working on the project, these things matter.

So why not handle it by pushing back into the model and doing something like

class Employee < ActiveRecord::Base
  def displayable?
    self.recently_hired? && self.full_time?
  end
end

With the above, we’ve now attached presentation logic to the model. Whether something is “displayable” should not be the concern of a model, especially when it’s not aware of the context the data is being used.

Fortunately, Rails Helpers allow us to pull anything that resembles domain logic out of the View completely but still influence what the user sees in the View:

module EmployeesHelper
  def display_employee(employee)
    if employee.recently_hired? && employee.full_time?
      content_tag(:div, "#{employee.full_name}, #{employee.hire_date}")
    end
  end
end

...as well as simplifying the View:

123	<% div_for employee do %> <%= display_employee(employee) %><% end %>

Several use cases, in which helpers are handy for evaluating a model and generating presentation logic, include:

Dynamically inserting CSS selectors into markup
Evaluating whether text should be hyperlinked
Changing the layout (markup) of the screen

Amazon Web Services EC2 Image

2008-06-18T22:57:00Z

If you ever dabble with Amazon’s Elastic Compute Cloud and need a reliable image prepackaged with Ruby on Rails, Nginx, and more, you can download our public AMI.

Improving Performance in Ruby on Rails Models

2008-06-11T08:00:00Z

The Active Record design pattern simplifies data access in Rails applications. But you can also shoot yourself in the foot performance-wise if you misapply it to queries that span multiple objects and instances. In this article, we look at ways you can improve query performance.

A Good Measure

So where does one start? As a rule of thumb, always optimize the slowest queries first. It sounds obvious but you’d be surprised. Use the application log to get a general sense of which queries are taking the longest. You can also apply the “benchmark” method—available in several key places:

Inside models:

def self.get_active
  self.benchmark 'Get Active Contacts' do 
    find_all_by_status('active')
  end
end

Inside controllers (acting upon a model):

123	Contact.benchmark 'Get Active Contacts' do @active_contacts = Contact.get_activeend

Inside views:

123	<% benchmark 'Show Active Contacts' do %> <%= @active_contacts %><% end %>

A cool feature is the ability to string multiple benchmarks together as follows:

Contact.benchmark 'First Way' do
  @active_contacts = Contact.get_active_test1
end

Contact.benchmark 'Second Way' do
  @active_contacts = Contact.get_active_test2
end

Contact.benchmark 'Third Way' do
  @active_contacts = Contact.get_active_test3
end

which then appears in your app log like this

123	First Way (0.65363)Second Way (0.56675)Third Way (0.54476)

Beyond these basics, you can explore the many profiling tools available to Ruby programmers.

Including Data Upfront

Often, you need to display a model instance and its related associations. Example: a company view also displays a list of related contacts. Active Record permits you to retrieve as much data as you need using the ‘include’ symbol; this is often referred to as ‘eager loading of associations.’

123	companies = Company.find(:all, :include => :contacts, :conditions => "contacts.status = 'Active'")

Behind the scenes, the SQL statement will include a join to the related table(s).

However, the previous example does not always result in better performance. While you can significantly cut down on the number of roundtrips to the database, the amount of data returned, to be processed by Ruby code and wrapped into an Active Record object, can in fact slow things down. This is especially true if your model contains a lot of attributes.

One way to speed things up is to only load the columns needed to support the view. Going back to the company example, we only care about showing contacts’ names and email addresses.

companies = Company.find(:all,
                         :include => :contacts, 
                         :select => "companies.*, contacts.first_name, 
                                     contacts.last_name, contacts.email" 
                         :conditions => "contacts.status = 'Active'")

A Little Bit of Ruby Love

Depending on the size of your database, sometimes it’s faster to retrieve ALL of the rows for a given model and then use Ruby’s sophisticated array handling capabilities to parse through the collection.

For example, let’s say we’re generating a report that counts the number of active contacts for all companies in the system. As the number of contacts grow, the following snippet will result in longer execution time. The ‘find_all_by_status’ method results in another database call for each company instance:

123	Company.find(:all).each do \|company\| puts company.contacts.find_all_by_status('active')end

The following alternative gets us down to two database calls:

1234	active_contacts = Contact.find_all_by_status('active')Company.find(:all).each do \|company\| puts active_contacts.select{ \|contact\| contact.company_id == company.id }end

What about the :include symbol? In some of our data access tests, the above variation proved to be faster than eagerly loading associations. Our guess is the construction of the larger SQL statement is a bit slower than two simple “select *” calls. Again, use the Benchmark functions to see for yourself. Only testing will help you come to your own conclusions.

Are You (De)Normal?

If you have a summary value that is expensive to calculate (i.e. requires multiple database roundtrips and/or a complex block of Ruby code) and changes frequently enough where caching won’t help much, then denormalizing the attribute may be useful.

Let’s say you are calculating a list of top scores of participants in a series of games. One way to do it is

123	Player.find(:all).each do \|player\| puts player.games.sum{ \|game\| game.score }end

If you only have a handful of players and games in the system, you’re fine. But increase it 10-fold and your nice “top scores” view begins to drag since a database call is being made to get each game’s score. Wouldn’t it be great if we could consolidate things to one roundtrip, like this:

123	Player.find(:all, :order => 'total_score DESC').each do \|player\| puts player.total_scoreend

Well, you can, with a little bit of refactoring and applying a Ruby on Rails observer object. Observers allow you to attach behaviors to specific model events, or callbacks. For example, you could write code that automatically logs an entry to a database table or sends out an email notification when a specific type of model is saved.

First, add the summary field – in this example, the ‘total_score’ – to the desired table using a Rails migration script.

Then, create an observer object that will “watch” Game instances. Any time a Game’s score is updated, we will summarize the player’s total score in their record.

Creating the observer is easy, just use the generate script:

script/generate observer game

Inside the new GameObserver class, add this:

class GameObserver < ActiveRecord::Observer
  def after_save(game)
    player = game.player
    player.update_attribute('total_score', player.total_score + game.score)
  end
end

But wait – you’re not done yet – you’ll also need to update your application’s environment.rb to activate the observer:

1234	Rails::Initializer.run do \|config\| #other config settings go here config.active_record.observers = :game_observerend

Then be sure you update any code still using the real-time calculated version of player’s scores and replace with player.total_score. If you’re trying to get the top 10 players, it’s as simple as

Player.find(:all, :order => 'total_score DESC', :limit => 10)

Rolling Up the Sleeves

In some cases, writing your own SQL can be the solution to squeezing out every millisecond of query performance. The trade-off, of course, is code maintenance. Every time you add a new field to your model, you’ll have to remember to update any hand-written queries. Everything you can do with SQL is abstracted in Active Record, so you don’t need to resort to it often. However, it can come in handy in reporting scenarios, where your query becomes complex, needing to display attributes from many objects, sometimes indirectly associated.

For example: display a list of emails for all active contacts associated with clients in the ‘West’ sales region, who have open orders, containing items of category X with a status of ‘Backordered.’

Contact.find_by_sql("select email from contacts co, clients cl, regions r, 
orders o, line_items li, products p, product_categories pc where 
co.client_id = cl.id and cl.region_id = r.id and o.client_id = cl.id 
and p.order_id = o.id and li.order_id = o.id and p.category_id = pc.id 
and r.region = 'West' and o.status = 'Open' and pc.name = 'X' and 
li.status = 'Backordered'")

Tuning the Backend

Adding a database index to a table, especially on columns that are frequently involved in joins, can boost performance as well. But be careful, as creating an index will improve read operations, but on the flip side, write operations will slow down. This article won’t go into the intricacies of database indexing, since this topic is already covered in detail elsewhere. However, Rails does makes things easy through migration scripts.

class AddGamesIndexes < ActiveRecord::Migration
  def self.up
    add_index :games, :player_id
  end

  def self.down
    remove_index :games, :player_id
  end
end

What Next

We discussed some techniques that help you measure and refine performance in Ruby on Rails models before needing to consider other options, such as caching.

We also recommend keeping your Rails framework current and peeking into Edge Rails every now and then. Over the past 6 months, for instance, there has been a lot of activity focused on Active Record performance optimizations. Rails 2.0 introduced query caching. This is an area which will continue to improve.

Congratulations to Sapphire Energy

2008-05-29T01:00:00Z

Today, Sapphire Energy launched their public Web site. We are happy to have been part of this effort, which started at the beginning of the year and saw us join forces with Edelman, a global PR firm known for its great achievements and merits.

Sapphire Energy’s ground-breaking technology allows the production of gasoline from algae. This isn’t biodiesel either—it’s 91 octane gas that can be used in your car. In addition, Sapphire Energy does not need to use agricultural land as a source for its fuel. To learn more, check out Forbes’ write-up.

How Webficient Helped

We were primarily responsible for the implementation of the Web site you see today. One of the highlights involved creating a content management system for site administrators and marketing folks, so that items such as press releases and news clips could be easily managed. Supporting Edelman, we ensured the final product looked as good as the designer’s prototypes and the content was accurate.

Supporting Roles

Behind the scenes, there’s a lot more involved with a project of this caliber. To preemptively block domain squatters from buying up variations of SapphireEnergy.com, we helped research and secure dozens of other domain names.

We were responsible for all technology decisions, such as planning the infrastructure powering the Web site.

Also, a critical aspect of the project was preparing for launch day and the waves of visitors that would arrive in response to coverage in the major news outlets. We setup monitoring systems to detect any potential problems. Prior to the launch, we simulated various scenarios of site usage to test limits. This gave us a working model for understanding what to do as usage increased. [Editor’s note, 6/8/08: we’re happy to report that the site has been running since launch day without any downtime.]

Technology

We used the Ruby on Rails application framework to build what you see today. There are many reasons we love using Rails again and again in our projects, for clients and internal tools. We’ll elaborate on some of reasons in a more technical write-up in the near future.

Thanks

Webficient would like to thank its partners and vendors for contributing to the success of the overall project, launch day, and beyond.

Sapphire Energy’s design is the brilliant production of Paula Wood. This was our first opportunity to work with Paula, and we hope to do so again in the future.

Slicehost.com. Straightforward. Affordable. Awesome Web hosting. No problems to report since we started using their services earlier this year. We continue to recommend their services to clients and colleagues.

Fiveruns, and their server and application monitoring service, is one of our recent favorites. Their service is easy to setup and provides valuable metrics that system administrators will eat up. We were able to use Fiveruns to provide real-time numbers to our client. It’s a great complement to Google Analytics.

Who is Webficient?

2008-05-26T17:00:00Z

We are a group of IT professionals who love working on complex problems. We’ll be featuring some of our work here as case studies. We’ll be sharing our insights through blog posts, code snippets, and screenshots. We hope we can spark some interesting conversations. Thus, one goal for Webficient is to share a repository of knowledge that can be applied to your projects.

The other aspect, of course, is the business end. We’ve recently relaunched this site to be more than just a blog… we’re hoping to connect with new businesses and partners who want to work on big ideas. We’re also here to help you with the small ones too. :)

Fixing Mongrel's Page Caching

2008-02-04T04:58:00Z

While playing around with Rails page caching, we discovered a peculiar quirk. Any pages set to be cached are saved as static HTML files, but Mongrel doesn’t seem to be able to find them and continues to process subsequent requests dynamically. This only happens though if you change the Rails’ page cache directory in your application configuration. Upon digging further, here’s what we uncovered.

By default, Rails will cache files in /public, and this works fine if you enable caching in your development environment. However, with a different cache location, such as

config.action_controller.page_cache_directory = RAILS_ROOT + "/tmp/cache"

things no longer work as expected. It turns out that Mongrel completely ignores this setting when looking for cached content. Furthermore, Mongrel’s directory handler assumes that files will always be located under /public.

If you want to patch this up on your local machine, just apply this Mongrel patch.

Page Caching Tips

Here’s some useful info if you’re implementing page caching in Ruby on Rails.

Using Mongrel to read cached pages is fine for development and POC’s but your best bet is to put this responsibility into the hands of an HTTP front-end such as Apache or Nginx.

If you want to cache every view in a controller, instead of adding “caches_page” for each action, just add this line near the top of your controller:

after_filter { |c| c.cache_page}

If you’re caching pages in a password protected part of your site, a relatively painless solution is to configure Apache or Nginx to read an htpasswd file and handle authentication on the front-end. Alternately, you could use Rails 2.0’s basic authentication as part of a before_filter but this would only work with fragment caching (since full page caching ignores any controller filters completely).

Mac and SSH Keys

2007-11-16T22:48:00Z

If you’re Web programming on a Mac and are moving data around using the SSH network protocol, it’s a good idea to setup an SSH key on the destination server. You’ll speed up your workflow by not having to enter a password each time.

Setting up a key is pretty easy. First create the key on your machine using Terminal:

ssh-keygen -t rsa -f ~/.ssh/id_rsa -C "me@mymachine"

Then, run this command to create a special directory on your server that is needed for SSH key authentication. The username must be the one you’re going to use for SVN, Capistrano, etc. You can have a key for each user, e.g. deploy.

ssh username@subdomain.host.com 'mkdir ~/.ssh;chmod 700 ~/.ssh'

Then deploy the key to your server using the username you want to associate with the key:

scp ~/.ssh/id_rsa.pub username@subdomain.host.com:~/.ssh/authorized_keys2

Then update the permissions on the server for the authorized_keys file you just created:

ssh username@subdomain.host.com 'chmod 600 ~/.ssh/authorized_keys2'

Note: the above commands assume your destination machine is using the more secure, SSH2. For SSH, just change ‘authorized_keys2’ to ‘authorized_keys’ above.

You’re done. To test everything is working, you can simply ssh username@host via Terminal. If you’re still getting a password prompt, then you may have missed something.

Installing the RMagick gem on Mac OSX Leopard

2007-11-16T05:46:00Z

If you’re having problems installing the RMagick gem on Leopard and you’re not using MacPorts, this info may help.

First, ensure you have the latest version of the UNIX developer tools from your Leopard CD. Otherwise, you’ll see errors like

1234	ERROR: Failed to build gem native extension.configure: error: C compiler cannot create executablesSee `config.log' for more details.RMagick configuration failed with status 77.

Then, download the RMagick OSX Installer available here. It’s a Ruby script which retrieves all of the required dependencies and then builds them and RMagick.

Finally, install the gem:

sudo gem install rmagick

How to Configure Swiftiply with Nginx

2007-08-20T22:33:00Z

Swiftiply is a proxying solution which boosts the performance of Ruby on Rails and Merb apps. One configuration allows you to use Swiftiply as an HTTP proxy, leveraging what are known as “Swiftiplied” Mongrels. These patched Mongrels become clients of the Swiftiply server and maintain a persistent connection, which is a reversal of the usual proxying solution (think Pound, Squid, etc.).

Nginx is a lightweight, fast HTTP server, which is gaining popularity. Nginx is setup as a reverse proxy to Swiftiply, which in turn has a direct connection to multiple Mongrel processes.

So when an HTTP request comes in, the pipeline looks like this…

HTTP request -> Nginx -> Swiftiply -> Mongrels

To successfully make Nginx and Swiftiply play nicely with one another, the configuration is fairly straightforward.

In nginx.conf, the cluster consists of ONE server entry:

123	upstream mongrel_test_cluster { server 127.0.0.1:4000;}

This may appear strange at first glance but really, what you’re telling Nginx is to proxy all requests to the same port being listened on by Swiftiply, which in turn, communicates with the Mongrels.

In swiftiply.yml:

cluster_address: 127.0.0.1
cluster_port: 4000
daemonize: true
epoll: true
epoll_descriptors: 8192
map:
- incoming: localhost
outgoing: 127.0.0.1:5000
default: true
docroot: /usr/local/httpd/testapp
redeployable: true

Swiftiply is listening on port 4000 and passing requests to port 5000 (to the “Swiftiplied” Mongrels).

To transform your Mongrel into a Swiftiplied Mongrel, just add the following line to your app’s config file:

require 'swiftcore/swiftiplied_mongrel'

Then, it’s as simple as firing up your Swiftiplied Mongrels and the Swiftiply server (both are covered in more detail in the Swiftcore docs), and Nginx.

The other Swiftiply configuration, using “Evented” Mongrels, is much simpler and does not use the Swiftiply server.

Just add this include to your Web application’s configuration file:

require 'swiftcore/swiftiplied_mongrel'

and configure Nginx as you would with a standard set of Mongrels.

upstream mongrel_test_cluster {
  server 127.0.0.1:4000;
  server 127.0.0.1:4001;
  server 127.0.0.1:4002;
  server 127.0.0.1:4003;
}

Then start up your Mongrels as described in the Swiftiply docs, and you’re done.