# Article on screen scraping w HTree+REXML, RubyfulSoup, WWW::Mechanize **URL:** \ **Category:** ruby-talk\ **Created:** [14 June 2006 10:07 UTC](https://rubytalk.org/t/article-on-screen-scraping-w-htree-rexml-rubyfulsoup-www-mechanize/28200 "2006-06-14T10:07:39Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Peter\_Szinek3](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@Peter\_Szinek3](https://rubytalk.org/u/Peter_Szinek3)\ **Post date:** [14 June 2006 10:07 UTC](https://rubytalk.org/t/article-on-screen-scraping-w-htree-rexml-rubyfulsoup-www-mechanize/28200/1 "2006-06-14T10:07:39Z")
Hello all, I am investigating the possibilities of screen scraping/web extraction/ automated web navigation/wrapper generation in Ruby. I have been working with these technologies for several years, (unfortunately) in Java and partially C/C++ only. I came to know Ruby a few months ago and I am currently investigating the existing tools for the above tasks. Since i have the feeling that i am not alone (this topic is brought up regularly here, maybe not as often as the "how to create an Object from it's name", but it is close to that 馃槈 I have summarized my findings (tools that i have found, descriptions, examples, comparison etc.), maybe can help someone. [http://www.rubyrailways.com/data-extraction-for-web-20-screen-scraping-in-rubyrails/](http://www.rubyrailways.com/data-extraction-for-web-20-screen-scraping-in-rubyrails/) You can find simple example solutions of the same problem (scraping links from a google result page) with regular expressions, HTree+REXML, RubyfulSoup and WWW::Mechanize. I am planning to write more entries on this topic, involving screen scraping from Rails, Gecko to Ruby GTK widget embedding, wrapper generation etc. Please note that i am new to Ruby so it is possible that my code snippets are not the most optimal yet (suggestions welcome), but they are all tested and working. Feedback/corrections/suggestions would be very much appreciated! If you liked the story, you can digg it here: [http://www.digg.com/programming/Data\_extraction\_for\_Web\_2.0:\_Screen\_scraping\_in\_Ruby\_Rails](http://www.digg.com/programming/Data_extraction_for_Web_2.0:_Screen_scraping_in_Ruby_Rails) Cheers, Peter ---
**Author:** ![James\_Edward\_Gray\_II](https://avatars.discourse-cdn.com/v4/letter/j/ea5d25/32.png) [@James\_Edward\_Gray\_II](https://rubytalk.org/u/James_Edward_Gray_II)\ **Post date:** [14 June 2006 12:13 UTC](https://rubytalk.org/t/article-on-screen-scraping-w-htree-rexml-rubyfulsoup-www-mechanize/28200/2 "2006-06-14T12:13:16Z")
> [http://www.rubyrailways.com/data-extraction-for-web-20-screen-scraping-in-rubyrails/](http://www.rubyrailways.com/data-extraction-for-web-20-screen-scraping-in-rubyrails/) This was a very good article. Thank you for sharing it with us. > Please note that i am new to Ruby so it is possible that > my code snippets are not the most optimal yet (suggestions welcome), Well, you sometimes declare variables inThisStyle, but Rubyists use this\_style\_here. James Edward Gray II > **路路路** > > On Jun 14, 2006, at 5:07 AM, Peter Szinek wrote: ---
**Author:** ![Peter\_Szinek3](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@Peter\_Szinek3](https://rubytalk.org/u/Peter_Szinek3)\ **Post date:** [14 June 2006 14:00 UTC](https://rubytalk.org/t/article-on-screen-scraping-w-htree-rexml-rubyfulsoup-www-mechanize/28200/3 "2006-06-14T14:00:52Z")
James Edward Gray II wrote: > > [http://www.rubyrailways.com/data-extraction-for-web-20-screen-scraping-in-rubyrails/](http://www.rubyrailways.com/data-extraction-for-web-20-screen-scraping-in-rubyrails/) > > This was a very good article. Thank you for sharing it with us. Thx! > Well, you sometimes declare variables inThisStyle, but Rubyists use > this\_style\_here. Thanks for the suggestion, i'll update it ASAP. (Coming from the Java camp, that's why the camelsAreStillHaunting 馃槈 Peter > **路路路** > > > On Jun 14, 2006, at 5:07 AM, Peter Szinek wrote: