Does anyone know of a web spider built in Ruby?
I couldn't find one either, so I wrote one based on http-access2 &
htmltokenizer (on RAA). Its really domain-specific though, I'm not
sure the code would be that useful to you.
I could send you the code if you want to get an idea of whats
involved, but be warned its really ugly (needed it fast).
Leon
···
On 5/4/05, Lucas Carlson <lucas@rufy.com> wrote:
Does anyone know of a web spider built in Ruby?
Here's an interesting link:
http://www.linux-magazine.com/issue/51/Ruby_Web_Spiders.pdf
It's not a full-blown lib, but seems to be a pretty good tutorial for
it. Regular expressions and simple http fetch methods make this sort
of thing fairly easy to do in Ruby.
···
On 5/4/05, Lucas Carlson <lucas@rufy.com> wrote:
Does anyone know of a web spider built in Ruby?