Here’s an example patch which makes “:disable_with” play nicely with the submit_tag when a remote form tag is used*.

module ActionView
  module Helpers
    module FormTagHelper
 
      def submit_tag(value = "Save changes", options = {})
        options.stringify_keys!
 
        if disable_with = options.delete("disable_with")
          disable_with = "this.value='#{disable_with}'"
          disable_with << ";#{options.delete('onclick')}" if options['onclick']
          disable_with << ";result = (this.form.onsubmit ? (this.form.onsubmit() ? this.form.submit() : false) : this.form.submit())"
          disable_with << ";if (result == false && this.form.onsubmit && (this.form.onsubmit+'').indexOf('Ajax.Request') == -1) { this.value = this.getAttribute('originalValue'); this.disabled = false }"
          disable_with << ";return result"
 
          options["onclick"] = [
            "this.setAttribute('originalValue', this.value)",
            "this.disabled=true",
            disable_with
          ].join(";")
        end
 
        if confirm = options.delete("confirm")
          options["onclick"] ||= ''
          options["onclick"] += "return #{confirm_javascript_function(confirm)};"
        end
 
        tag :input, { "type" => "submit", "name" => "commit", "value" => value }.update(options.stringify_keys)
      end
 
    end
  end
end

Be sure you restart your Rails app in order to see your patch take effect.

The above example overrides the existing Rails framework helper. However, you can also use the same technique to extend methods via alias_method_chain.

*The Rails remote_form_for tag adds a ‘return false’ to the form’s onsubmit event handler. However, the Rails submit_tag helper evaluates whether false was returned via onsubmit and will revert back to the original text (and re-enable the button) if this is the case. This makes sense when you are validating a form… the validation fails and you need everything reset so that you can correct your error and resubmit the form. However, for remote forms, in which an AJAX call is made to submit the form, we want the RJS response to drive the form’s (and page’s) behavior. So the patch evaluates whether an AJAX call is made during onsubmit; if so, it lets the submit button stay disabled. Hacky, I know. But it allows us to do what we need without having to touch the PrototypeHelper module. I tested with normal form_tag’s that use disable_with and it works in both cases.

Usually the above is not even a concern until you really do need your local application to handle more than one request at a time. Let me illustrate with an example. We’re working on an app in which an HTTP request “simulator” (within the same app) is used for testing inbound API requests. The code looks like this:

payload = 'stuff'
http = Net::HTTP.new('localhost',3000)
response = http.post2('/receiver', payload, {'Content-Type' => 'application/xml'})

and should post XML to http://localhost:3000/receiver. However, if you try running it by calling up a page in your browser, your request hangs and eventually times out. This is because the Rails dispatcher (through Mongrel) cannot process a second HTTP request - the XML post - because the first one - the browser request - is still running.

On aside, if you wrap the above into a new thread, you’ll see it does process immediately, e.g.

Thread.new {
  payload = 'stuff'
  http = Net::HTTP.new('localhost',3000)
  response = http.post2('/receiver', payload, {'Content-Type' => 'application/xml'})
}

but the last thing you want to do is introduce Ruby threading code into your non-thread safe Rails app (at least prior to version 2.2, which is still in ‘edge’ Rails as of this publish date).

Passenger to the Rescue

Before we go on, let me point out the lovely Passenger architectural overview, which explains how Apache prefork MPM is leveraged to spawn multiple instances of your Rails app. Each instance runs as a separate process but an internal application pool is used to conserve handles. The net result is that now your local Rails app will fork a new process for the next incoming request (while the first one is still running), and any code similar to the first example executes promptly. I want to emphasize that this is similar to having a pack of Mongrels on standby, each one ready to handle a new request. However, the overhead will be substantially less, especially if you use Ruby Enterprise Edition, also from the creators of Passenger.

Passenger on Mac OS X Leopard

Running Passenger on the Mac has been covered in detail in “Installing Phusion’s Passenger on Leopard (OS X 10.5),” a recommended step-by-step walk-through. You won’t need to compile Apache if you’re running 10.5 and a recent version of the Passenger gem. Overall, the configuration is quite easy, especially if you’ve dealt with Apache before.

The only gotcha’s are remembering to explicitly set the environment to ‘development’ in the virtual host settings in your Apache config file:

  DocumentRoot /httpd/spyglass/public
  ServerName spyglass.local
  RailsEnv development

…and adding an entry for your app in /etc/hosts, which we recommend if you work on multiple apps locally:

127.0.0.1 spyglass.local

Easy as Pie*

If you don’t want to deal with raw Apache settings or editing your hosts file, be sure to check out the Passenger preferences panel found here. It allows you to drag and drop your existing Rails app into a preferences window and will automatically configure all of the settings for you. The settings are persisted to a file located in /etc/apache2/passenger_pane_vhosts/ in case you want to tweak them manually.

*I like pie

Logging Caveat

While log entries will continue to appear under log/development.log, any Ruby-specific logging is dumped into Apache’s error log, /var/log/apache2/error.log (also accessible through the Mac Console). So for example, a “puts @some_variable” will be piped into Apache’s log. Therefore, if you are relying on any debug statements in your code, simply use logger.debug(’string’) instead, and it will appear in development.log.

Create a New Environment

Rails allows you to define new environments without much effort. Simply save a new configuration file to config/environments (example: development-phil.rb) with your custom settings. Then in database.yml, you’ll need to add a section that matches the name you used (development-phil).

When you start your Mongrel, just pass in your new environment name:

script/server --e=development-phil

The only caveat with this approach is to be aware of any code in your app that evaluates whether the current environment is ‘development.’ Assuming you’re using a naming convention of development-mycustomname.rb for your environment, simply swap out any instance of this code:

if RAILS_ENV == 'development'

with this:

if RAILS_ENV.include? 'development'

Initializers

Rails 2 improved configuration management by allowing you to break apart start-up settings into individual files. These settings are stored in config/initializers and are processed after environment.rb (and development.rb) are loaded.

If you only have a couple settings you want to localize (e.g. SMTP server), you may not want to spend the effort defining a new development environment as described above. Instead, create a new initializer file (example: mysettings.rb) but do not add it to source control. Note: both Subversion and Git allow you to set ‘ignore’ properties on specific files in case you are worried about accidentally checking these in (links below).

An initializer can be used to override existing settings in environment.rb and development.rb. The app may have a set of global settings common to all developers but within the initializer you can redefine them as needed.

If you work on multiple projects and are always re-applying the same local settings to your Rails apps, you can externalize your initializer in a common location and then create a symbolic link inside your Rails app. Just save your mysettings.rb outside of your Rails path and from within the console, navigate to your app’s config/initializers folder and type:

ln -s /apps/mysettings.rb

Additional Resources

How to ignore specific files during source code repository check-ins:
Git: .gitignore
Subversion: propset/propedit

If you work on a Mac, then a virtualization solution like Parallels (currently what we’re using) or VMWare Fusion (also highly recommended by colleagues) is your best bet for running Windows.

The next challenge is being able to run IE6 side by side with IE7 (or the newer IE8 beta). We’ve explored various tools over the last couple weeks but the one we settled on finally is TredoSoft’s IE Installer. Note, however, this installer is optimized for Windows XP, which represents our operating system of choice due to lower memory requirements than Vista (it also has a 74% market share as of August 2008). If you are looking for a solution for running IE6 under Vista, check out this article.

TredoSoft installs a fully functional version of IE6. There are some limitations, e.g. not being able to use the IE Developer Toolbar. And others have encountered some issues (read the comments on the bottom of the TredoSoft IE6 page). However, we’ve found this solution to be more stable than some of the others we’ve evaluated.

Congrats!

I’ve had the pleasure of working with Chris and Michael (and their team) on the Webvanta content publishing platform over the last several months. Today, they are unveiling their next generation CMS at the Techcrunch 50.

 Webvanta enables Web and graphics designers to easily create full featured Web sites without having to do any programming. You can create and categorize all types of content, configure a variety of features, customize the look and feel, and more. To see an example of an actual Webvanta-powered site, check out Spartina.

If you’re interested in becoming a beta tester, please sign up on their home page.

Behind the scenes, Webvanta is a sophisticated Ruby on Rails powered application hosted at EngineYard.

The Webvanta project is an example of how my company can help you with your Web development needs. Early stage companies have a common pattern in which there are weeks during which a lot of hours are needed to meet a strategic goal. But in between these “sprints,” you may not need or want to incur the same level of costs. With our on-demand development services, you only pay for what you need, whether this is one day a month, one day a week, or one full month of development services. Contact me if you’d like to find out more.

Our needs were simple: create an XML sitemap without having to add any new controllers, model attributes, and so on. Since we already had a fully functional, deployed Web app, pointing our crawler to it and generating a map containing only valid, internal links made a lot of sense.

Thanks to Alastair’s code, we learned about the structure and mechanism for constructing a Google Sitemap, which involves using Rails’ Builder::XmlMarkup class.

Here’s the code for the revised crawler. Please refer to the original article for configuration details.

require 'mechanize'
 
class Crawler
 
  EXTENSIONS_IGNORED = %w[.csv .doc .docx .gif .jpg .jpeg .js .mp3 
    .mp4 .mpg .mpeg .pdf .png .ppt .rss .swf .txt .xls .xlsx .xml]
 
  PROTOCOLS_IGNORED = %w[feed ftp itms javascript mailto]
 
  def initialize(starting_url, credentials = nil, quiet_mode = false, sitemap = false, debug = false)
    @bad_pages = []  
    @agent = WWW::Mechanize.new
    @sitemap = sitemap
    @debug = debug
    @visited_pages = []
 
    if credentials
      creds = credentials.split(':')
      @agent.basic_auth(creds[0], creds[1])
    end
 
    @quiet_mode = quiet_mode
    @starting_url = starting_url
    @starting_url_domain = starting_url[/([a-z0-9-]+)\.([a-z.]+)/i]
    puts "domain: #{@starting_url_domain}" if @debug
    extract_and_call_urls(starting_url)
    generate_sitemap if @sitemap
  end
 
  def extract_and_call_urls(url)            
    #get page
    puts "#{@visited_pages.size+1} #{url}" unless @quiet_mode
    begin
      page = @agent.get(url)
    rescue => exception
      @bad_pages << url
      puts "error: #{url}, #{exception.message}"
      return
    end
 
    #for any content types we may have missed above, exit if content type is not html
    return if page.instance_of?(WWW::Mechanize::File) || page.content_type.index('text/html') == nil
 
    #add to array
    @visited_pages << url
 
    #get links found on page
    links = page.links
 
    #for each link, call the url if not in history
    links.each{ |link| extract_and_call_urls(link.href) unless 
      ignore_url?(link.href) || @visited_pages.include?(link.href) }
  end
 
  private
 
  def ignore_url?(url)
    begin
      return ignored = true if url.nil? ||
                       (url.include? 'http' and !url.include?("webficient.com")) ||
                       @bad_pages.include?(url) ||
                       PROTOCOLS_IGNORED.find{ |prt| url =~ /#{prt}:/ } != nil ||
                       EXTENSIONS_IGNORED.find{ |ext| url =~ /#{ext}$/ } != nil
    ensure
      puts "ignored: #{url}" if ignored and @debug
    end
  end
 
  def generate_sitemap
  	xml_str = ""
  	xml = Builder::XmlMarkup.new(:target => xml_str, :indent=>2)
 
  	xml.instruct!
  	xml.urlset(:xmlns=>'http://www.sitemaps.org/schemas/sitemap/0.9') {
  		@visited_pages.each do |url|
  		  unless @starting_url == url
    	    xml.url {
      	    xml.loc(@starting_url + url)
      			xml.lastmod(Time.now.utc.strftime("%Y-%m-%dT%H:%M:%S+00:00"))
      			xml.changefreq('weekly')
   			  }
   			end
  		end
  	}
 
  	save_file(xml_str)
  	update_google
  end
 
	# Saves the xml file to disc. This could also be used to ping the webmaster tools
	def save_file(xml)
		File.open(RAILS_ROOT + '/public/sitemap.xml', "w+") do |f|
			f.write(xml)	
		end		
	end
 
	# Notify google of the new sitemap
	def update_google
	    sitemap_uri = @starting_url + '/sitemap.xml'
	    escaped_sitemap_uri = URI.escape(sitemap_uri)
	    Net::HTTP.get('www.google.com',
	                  '/webmasters/sitemaps/ping?sitemap=' +
	                  escaped_sitemap_uri)
	end
 
end

What’s Going On

As the crawler hits each page, it persists a valid URL into an array:

@visited_pages << url

Then, we iterate through the array and build the XML:

xml.url {
  xml.loc(@starting_url + url)
  xml.lastmod(Time.now.utc.strftime("%Y-%m-%dT%H:%M:%S+00:00"))
  xml.changefreq('weekly')
}

Note that we are always setting the modification date of every page to the current time and anticipated frequency of page updates to ‘weekly.’ Feel free to vary or extrapolate into additional config settings.

Connecting it Together

In this iteration, we packaged this up as a Rake task by saving the following task as lib/tasks/admin.rake (inside our Rails application directory). If you want to run this from Capistrano instead, just add the task to config/deploy.rb.

require 'lib/crawler'
 
namespace :admin do
 
  desc "Crawl pages using the Mechanize gem. Set URL variable as a starting point. Set CREDS as username:password if you are hitting a password protected site. To generate a Google Sitemap in /public/sitemap.xml, set SITEMAP=true. To suppress output and only show errors, set QUIET=true. To show more details during output, set DEBUG=true."
  task :crawl_pages do
    start_url = ENV["URL"] || "http://localhost:3000"
    sitemap = Crawler.new(start_url, (ENV["CREDS"] if ENV["CREDS"]), ENV["QUIET"] || false, ENV["SITEMAP"] || false, ENV["DEBUG"] || false)
  end
 
end

Sitemap-ilicious!

Run your new rake task from the command line interface:

rake admin:crawl_pages URL=http://www.webficient.com SITEMAP=true

Google is automatically pinged as the final step. A copy of the sitemap is also saved to public/sitemap.xml. You can point Google’s Webmaster Tools to this URL (http://mywebsite.com/sitemap.xml) to track Google crawler statistics.

I recently came across a post which claims Craigslist’s fight against spam is becoming more challenging. New counter-spam techniques are tested, only to be defeated by spammers armed with increasingly sophisticated software and schemes.

Historically, the Craigslist community has been vigilant in keeping listings fair and honest by being able to ‘flag’ any post. Everyone can become a virtual watchdog and keep the muck out. But when the ‘community’ is infiltrated by bots, the game changes a bit. At some tipping point, it becomes extremely difficult for the good guys to outpace the bad ones. 

There are two ways to solve this problem. One involves advanced heuristic algorithms that require a lot of hardware, code, and money - very doable given their estimated revenue. But the other is likely cheaper and more plausible to work as the cat and mouse games continue to evolve: fight people with people.

Enter the Turk

A while back, Amazon rolled out a unique Web service known as Amazon Mechanical Turk (AMT). The idea was simple: take tasks that may be challenging or costly for computers to perform and farm it out to a virtual workforce of people who, in exchange for doing the task, get monetary compensation. 

The Mechanical Turk is a great fit for craigslist because in most cases, a human is better at determining whether a listing is spammy. A machine will have the advantage when looking for patterns across listings (e.g. duplicated mass listings with slight variations). But give a person a single page and within seconds a determination will be made.

Any developer can integrate an app with the Mechanical Turk through its API, with clients available for Java, Perl, Ruby, and more. A hypothetical craigslist integration could look like the following illustration:

Those Listings Add Up Quick

Even with a conservative cost of one cent per completed task, given the number of monthly listings (Googling it shows an estimated 30 million), craigslist’s bill from Amazon would be significant. Perhaps they could limit the use AMT to listings considered suspect based on their existing anti-spam algorithms. But I think Amazon should throw it in for free - well, there’s a catch of course.

How Are You Gonna Pay For This

One of the biggest things missing from craigslist is a payment platform. Low ticket items are usually paid with cash. But as we start getting up there, there’s definitely an area where cash and checks become a bit of a drag. And as we get further into m-commerce, being able to complete a transaction with your phone at a local garage sale visit will become the norm. There’s nothing preventing craigslist users from using PayPal or Google Checkout today but having an integrated option could make things easier.

A payment system can also be used to establish trust and reputation. Sure, there will always be some level of fraud, but it will be the lesser of two evils compared to today’s world of ‘buyers’ offering to pay with wire transfers. And I don’t believe a payment system will detract from the emphasis on ‘local’ community. People will continue to buy from others in their zip code; it’ll simply be a lot easier to pay for stuff.

Amazon’s Payment Systems is a relatively new entrant to the space. They are marketing it to developers to help proliferate it. Obviously an endorsement by someone like craigslist would be huge. Since Amazon gets a cut from each transaction, a nice revenue stream would arise if all of a sudden, 1 out of every 10 listings resulted in a payment transaction.  For this reason, Amazon could afford to give away Mechanical Turk for free and even fund the cost of each task payout.

What do you think? Would Amazon’s services make craigslist a better place?

SPF Defined

The Sender Policy Framework (SPF) was created to reduce spam by detecting impersonation of domain names. Since anyone can forge a “from” email address in a message header, SPF can be used by email servers to verify that the message originates from the domain it claims to be from, or that it was sent from a trusted host.

Creating a DNS TXT Record

SPF data is stored in a DNS TXT record, which is supported by most domain registrars. Simply go to the DNS management console used by your registrar or Web host and look for a “TXT” record option.

Let’s say you’re using a third-party service such as Google Apps to handle your corporate email. The TXT record would be

v=spf1 mx include:v=spf1 include:aspmx.googlemail.com ~all

But what about if you’re also sending emails directly from an application server? You’d want to include the IP address of the machine hosting the SMTP server:

v=spf1 a mx ip4:67.207.129.94 include:v=spf1 include:aspmx.googlemail.com ~all

To view a detailed explanation about the above parameters and an easy-to-use wizard for generating SPF-compliant DNS entries, check out the SPF Setup Wizard.

Before and After

If you examine your email headers, you can see the impact of creating an SFP record. Here’s an email sent to a GMail account before the record was added.

Received-Spf: neutral (google.com: 67.207.129.94 is neither permitted nor denied by best guess record for domain of info@webficient.com) client-ip=67.207.129.94;
Authentication-Results: mx.google.com; spf=neutral (google.com: 67.207.129.94 is neither permitted nor denied by best guess record for domain of info@webficient.com) smtp.mail=info@webficient.com

Now compare the headers with the SPF record in place:

Received-Spf: pass (google.com: domain of info@webficient.com designates 67.207.129.94 as permitted sender) client-ip=67.207.129.94;
Authentication-Results: mx.google.com; spf=pass (google.com: domain of info@webficient.com designates 67.207.129.94 as permitted sender) smtp.mail=info@webficient.com

Why Validate HTML?

For an introduction to validating your Web site, see the following resources:

• W3C’s “Why Validate?” covers the practical arguments.
• “HTML Validation: the hidden key to SEO” offers economic motives.

Convinced? Read on…

How to Validate Your Web Site

There are several ways to validate your Web site. Easiest: enter your Web site address into W3C validator service and see the results immediately.

Or if you’re a Web developer and want to do this on a regular basis, get the “Web Developer” add-on for Firefox, and you’ll find a “Validate Local HTML” option in the Tools section.

And for the Ruby on Rails programmers in the house, you can automate the validation process as part of  the test cycle using the nifty htmltest plug-in.

Fixing the Video Embed Code

Back to our original problem: YouTube video code wraps an EMBED tag inside an OBJECT tag but the former is not part of the XHTML standard. Hence, a Web page run through the validation service will fail. Getting rid of the EMBED tag requires a few changes to the OBJECT tag so that browsers like Firefox display the video properly.

<object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" width="425" height="344" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0"><param name="src" value="http://www.youtube.com/SOME_VIDEO_URL" /><embed type="application/x-shockwave-flash" width="425" height="344" src="http://www.youtube.com/SOME_VIDEO_URL"></embed></object>

The above code was tested in IE 7, Firefox 3, and Safari 3. Your mileage may vary with older browsers.

Update

Nick Pettit shared a delightful tool which solves the above headache and generates valid XHTML 1.0 Strict markup. Check out http://validifier.com.