# Webcrawler/robot

**URL:** <https://rubytalk.org/t/webcrawler-robot/22061>\
**Category:** ruby-talk\
**Created:** [1 November 2005 08:37 UTC](https://rubytalk.org/t/webcrawler-robot/22061 "2005-11-01T08:37:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mikkel\_Bruun](https://avatars.discourse-cdn.com/v4/letter/m/e56c9b/32.png) [@Mikkel\_Bruun](https://rubytalk.org/u/Mikkel_Bruun)\
**Post date:** [1 November 2005 08:37 UTC](https://rubytalk.org/t/webcrawler-robot/22061/1 "2005-11-01T08:37:07Z")

</div>

Hi Guys,

Ive been looking for a webcrawler/robot, implemented in ruby...

but I havent found any...can this really be true???

thanks in advance

Mikkel

---

<div class="post-metadata">

**Author:** ![Jay4](https://avatars.discourse-cdn.com/v4/letter/j/f1d935/32.png) [@Jay4](https://rubytalk.org/u/Jay4)\
**Post date:** [1 November 2005 16:07 UTC](https://rubytalk.org/t/webcrawler-robot/22061/2 "2005-11-01T16:07:06Z")

</div>

There's one in the Rails Analyzer project that crawls a single domain:

RDoc:  
[http://rails-analyzer.rubyforge.org/hacks/](http://rails-analyzer.rubyforge.org/hacks/)

Article:  
[http://groups.google.com/group/comp.lang.ruby/browse\_thread/thread/166fa4076defb055](http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/166fa4076defb055)

---

<div class="post-metadata">

**Author:** ![Gene\_Tani](https://avatars.discourse-cdn.com/v4/letter/g/4491bb/32.png) [@Gene\_Tani](https://rubytalk.org/u/Gene_Tani)\
**Post date:** [1 November 2005 21:52 UTC](https://rubytalk.org/t/webcrawler-robot/22061/3 "2005-11-01T21:52:08Z")

</div>

[http://rubyforge.org/snippet/detail.php?type=snippet&id=49](http://rubyforge.org/snippet/detail.php?type=snippet&id=49)

jfister@gmail.com wrote:

> **···**
>
> > There's one in the Rails Analyzer project that crawls a single domain:
> > 
> > RDoc:  
> > [http://rails-analyzer.rubyforge.org/hacks/](http://rails-analyzer.rubyforge.org/hacks/)
> > 
> > Article:  
> > [http://groups.google.com/group/comp.lang.ruby/browse\_thread/thread/166fa4076defb055](http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/166fa4076defb055)

---

<div class="post-metadata">

**Author:** ![Eric\_Hodel1](https://avatars.discourse-cdn.com/v4/letter/e/a9adbd/32.png) [@Eric\_Hodel1](https://rubytalk.org/u/Eric_Hodel1)\
**Post date:** [1 November 2005 22:42 UTC](https://rubytalk.org/t/webcrawler-robot/22061/4 "2005-11-01T22:42:30Z")

</div>

It is only useful as a benchmarking tool, it is too stupid to do any thing useful beyond timing.

> **···**
>
> On Nov 1, 2005, at 8:07 AM, jfister@gmail.com wrote:
> 
> > There's one in the Rails Analyzer project that crawls a single domain:
> > 
> > RDoc:  
> > [http://rails-analyzer.rubyforge.org/hacks/](http://rails-analyzer.rubyforge.org/hacks/)
> > 
> > Article:  
> > [http://groups.google.com/group/comp.lang.ruby/browse\_thread/thread/166fa4076defb055](http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/166fa4076defb055)
> 
> --  
> Eric Hodel - drbrain@segment7.net - [http://segment7.net](http://segment7.net)  
> FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04