Comments on: Generate a Google Sitemap in Rails

By: Sitemaps vía crawling

Sitemaps vía crawling — Thu, 16 Jul 2009 01:49:16 +0000

[...] encontré una solución práctica para este caso (donde hay pocas páginas, menos de 1k) que usa un crawler para recorrer [...]

By: Sitemaps vía crawling | El Futirifoken

Sitemaps vía crawling | El Futirifoken — Wed, 15 Jul 2009 17:36:36 +0000

[...] encontré una solución práctica para este caso (donde hay pocas páginas, menos de 1k) que usa un crawler para recorrer [...]

By: Chris Lerum

Chris Lerum — Wed, 20 May 2009 15:49:32 +0000

Thank you. Incredibly wonderful and useful. Worked great first time.

By: John Wells

John Wells — Sun, 22 Feb 2009 16:52:11 +0000

I’ve had a lot of challenges with urls that include paging. Often page links are placed in a relative way, like “?page=1″, on the current page. So you might have:

http://www.mysite.com/listings?page=1
and
http://www.mysite.com/pictures?page=1

They’re really two valid and separate pages, but the script you’ve written considers them the same, and won’t properly handle them. Just an FYI if someone else has a similar problem.

By: Phil Misiowiec

Phil Misiowiec — Thu, 05 Feb 2009 22:26:57 +0000

James,

We’re looking into this… it may be a fault with the Mechanize gem used by the script.

By: JamesD

JamesD — Thu, 05 Feb 2009 22:03:39 +0000

Any explanation as to why I’m getting errors like this? They are all valid requests..
rror: /users/cheappete, 404 => Net::HTTPNotFound
77 /items/22-travelodge-ireland-winter-sale-rooms-from-29-eur39
error: /items/22-travelodge-ireland-winter-sale-rooms-from-29-eur39, 404 => Net::HTTPNotFound
77 /items/22-travelodge-ireland-winter-sale-rooms-from-29-eur39/shares/new
error: /items/22-travelodge-ireland-winter-sale-rooms-from-29-eur39/shares/new, 404 => Net::HTTPNotFound
ignored: /items/22-travelodge-ireland-winter-sale-rooms-from-29-eur39

By: Phil Misiowiec

Phil Misiowiec — Thu, 29 Jan 2009 10:07:20 +0000

There are a few things we’ve tweaked that should help with some of the open items above, including memory usage. We’ll post an updated script once we’ve had a chance to test a couple larger apps. We’d like to package up as a gem at some point as well. Stay tuned…

By: JamesD

JamesD — Fri, 16 Jan 2009 23:17:33 +0000

Getting a lot of errors running the rake task on my server

e.g.
93 /items/106-business-ideas-book
error: /items/106-business-ideas-book, 404 => Net::HTTPNotFound

even through the URL is open http://www.goodbuy.ie/items/106-business-ideas-book

Any ideas?

By: pjam

pjam — Wed, 29 Oct 2008 03:31:14 +0000

Phil, just wanted to report, that i ran the sitemap and it was pulling 944mb of ram.

that is probably why i couldn’t run it for long on my shared host.

A suggestion for friends of the blog - run it on your home computer with lots of memory. i have a 4 gb mac, and it lasts longer. although now i get to find out why it didn’t build the map.

i got to page 3360 this time before it crapped out. is there a history limit phil?

By: Phil Misiowiec

Phil Misiowiec — Fri, 24 Oct 2008 22:33:28 +0000

The crawler script has been revised as of October 24, 2008. We plan on wrapping into a Ruby gem next week, pending some additional testing.