# Reading website **URL:** \ **Category:** ruby-talk\ **Created:** [17 August 2006 19:40 UTC](https://rubytalk.org/t/reading-website/30080 "2006-08-17T19:40:07Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![fabsy](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@fabsy](https://rubytalk.org/u/fabsy)\ **Post date:** [17 August 2006 19:40 UTC](https://rubytalk.org/t/reading-website/30080/1 "2006-08-17T19:40:07Z")
Hey! I have been going through some basic IO stuff and want to get into something more "advanced". I would like to make a script who asks for a website and takes the information (the text) on the website and put it into a textfile.. So now im wondering.. how do I tell the script to download/read text/content from a website.. How do I do it? ( of course i do not want anyone to actually write this script for me, i want to write it myself but im a newbie and don't know which libs and etc to read in) Thanks! ---
**Author:** ![James\_Edward\_Gray\_II](https://avatars.discourse-cdn.com/v4/letter/j/ea5d25/32.png) [@James\_Edward\_Gray\_II](https://rubytalk.org/u/James_Edward_Gray_II)\ **Post date:** [17 August 2006 19:47 UTC](https://rubytalk.org/t/reading-website/30080/2 "2006-08-17T19:47:09Z")
You are looking for the open-uri library. See if these docs are enough to get you going:    [http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html](http://www.ruby-doc.org/stdlib/libdoc/open-uri/rdoc/index.html) James Edward Gray II > **···** > > On Aug 17, 2006, at 2:40 PM, fabsy wrote: > > > Hey! > > I have been going through some basic IO stuff and want to get into > > something more "advanced". > > I would like to make a script who asks for a website and takes the > > information (the text) on the website and put it into a textfile.. > > > > So now im wondering.. how do I tell the script to download/read > > text/content from a website.. How do I do it? ( of course i do not want > > anyone to actually write this script for me, i want to write it myself > > but im a newbie and don't know which libs and etc to read in) ---
**Author:** ![Patrick\_Hurley1](https://avatars.discourse-cdn.com/v4/letter/p/b77776/32.png) [@Patrick\_Hurley1](https://rubytalk.org/u/Patrick_Hurley1)\ **Post date:** [17 August 2006 19:48 UTC](https://rubytalk.org/t/reading-website/30080/3 "2006-08-17T19:48:04Z")
Just require "open-uri" open(url) do |io|   # the ensues end > **···** > > On 8/17/06, fabsy \ wrote: > > > Hey! > > I have been going through some basic IO stuff and want to get into > > something more "advanced". > > I would like to make a script who asks for a website and takes the > > information (the text) on the website and put it into a textfile.. > > > > So now im wondering.. how do I tell the script to download/read > > text/content from a website.. How do I do it? ( of course i do not want > > anyone to actually write this script for me, i want to write it myself > > but im a newbie and don't know which libs and etc to read in) > > > > Thanks! ---
**Author:** ![Chris6](https://avatars.discourse-cdn.com/v4/letter/c/b2d939/32.png) [@Chris6](https://rubytalk.org/u/Chris6)\ **Post date:** [17 August 2006 19:55 UTC](https://rubytalk.org/t/reading-website/30080/4 "2006-08-17T19:55:11Z")
fabsy wrote: > Hey! > I have been going through some basic IO stuff and want to get into > something more "advanced". > I would like to make a script who asks for a website and takes the > information (the text) on the website and put it into a textfile.. Have a look at [Programming Ruby: The Pragmatic Programmer's Guide](http://ruby-doc.org/docs/ProgrammingRuby/), the section titled 'Network and Web', for the low level details. Open-uri is a nice wrapper to make web IO simple as File IO Look into Mechanize/Hpricot/etc if you want to get fancy 9^) Cheers ---
**Author:** ![Mat\_Schaffer](https://avatars.discourse-cdn.com/v4/letter/m/71e660/32.png) [@Mat\_Schaffer](https://rubytalk.org/u/Mat_Schaffer)\ **Post date:** [18 August 2006 03:09 UTC](https://rubytalk.org/t/reading-website/30080/5 "2006-08-18T03:09:53Z")
I'm a little late to this party, and it might be a little advanced for you yet, but if you're dealing with scripting against websites you can do some pretty sick stuff using 'mechanize' ([http://mechanize.rubyforge.org/\](http://mechanize.rubyforge.org/%5C)) and 'hpricot' ([http://code.whytheluckystiff.net/hpricot/\](http://code.whytheluckystiff.net/hpricot/%5C)). Mechanize can click links and fill out forms. Hpricot can look for text/tags in the resulting HTML. And Mechanize supports pluggable parsers which lets you drop Hpricot right into Mechanize. It's so buttery... 🙂 -Mat > **···** > > On Aug 17, 2006, at 3:40 PM, fabsy wrote: > > > Hey! > > I have been going through some basic IO stuff and want to get into > > something more "advanced". > > I would like to make a script who asks for a website and takes the > > information (the text) on the website and put it into a textfile.. > > > > So now im wondering.. how do I tell the script to download/read > > text/content from a website.. How do I do it? ( of course i do not want > > anyone to actually write this script for me, i want to write it myself > > but im a newbie and don't know which libs and etc to read in) > > > > Thanks! ---
**Author:** ![fabsy](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@fabsy](https://rubytalk.org/u/fabsy)\ **Post date:** [17 August 2006 20:15 UTC](https://rubytalk.org/t/reading-website/30080/6 "2006-08-17T20:15:10Z")
Thanks! I've got it working.. > **···** > > --- > require "open-uri" > > print 'Skriv in adress: ' > addr = gets.chomp > >   open(addr) do |addr| >     fil = File.open('file', 'a') >   fil \<\< addr.read >   fil.close > end > --- > > Is that a good way to write it? > I also saw that the first "word" in the file 'file' was > #\.. What does that mean? ---
**Author:** ![fabsy](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@fabsy](https://rubytalk.org/u/fabsy)\ **Post date:** [17 August 2006 20:35 UTC](https://rubytalk.org/t/reading-website/30080/7 "2006-08-17T20:35:05Z")
Thanks! I've got it working..Is this a good way to write it? > **···** > > --- > require "open-uri" > > print 'Skriv in adress: ' > addr = gets.chomp > >         open(addr) do |addr| >         fil = File.open('file', 'a') >         fil \<\< addr.read >         fil.close > end > --- ---
**Author:** ![David\_Vallner](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@David\_Vallner](https://rubytalk.org/u/David_Vallner)\ **Post date:** [17 August 2006 21:25 UTC](https://rubytalk.org/t/reading-website/30080/8 "2006-08-17T21:25:08Z")
It means you wrote an input stream object into the file. Which seems Weird. Wasn't there anything in 'file' before this script started? You set it to append, so that might be a leftover from some other script you wrote that used that file. Delete it, or open the file with the 'w' flag instead of 'a', and run it again? \ Also, you might want to avoid open(addr) {|addr| ... }, reusing the same variable name. It's probably not the issue here, but shadowing variables is inherently confusing. YMMV. \ David Vallner > **···** > > On Thu, 17 Aug 2006 22:15:10 +0200, fabsy \ wrote: > > > Thanks! > > I've got it working.. > > > > --- > > require "open-uri" > > > > print 'Skriv in adress: ' > > addr = gets.chomp > > > >   open(addr) do |addr| > >     fil = File.open('file', 'a') > >   fil \<\< addr.read > >   fil.close > > end > > --- > > > > Is that a good way to write it? > > I also saw that the first "word" in the file 'file' was > > #\.. What does that mean? ---
**Author:** ![fabsy](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@fabsy](https://rubytalk.org/u/fabsy)\ **Post date:** [17 August 2006 21:55 UTC](https://rubytalk.org/t/reading-website/30080/9 "2006-08-17T21:55:05Z")
Aha! Yes it was something old. It opened the file with append.. And, sorry for all the questions.. (im really trying to learn) 🙂 But is there any thing like this in ruby? --VB code-- If InStr(1, Text1.Text, "String to find", 1) \> 0 Then Msgbox "I found the string!" > **···** > > -------- > > ( If InStr(Start,String1,String2,CompareMode)\>0 Then Msgbox "I found > the string!") ---
**Author:** ![David\_Vallner](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@David\_Vallner](https://rubytalk.org/u/David_Vallner)\ **Post date:** [17 August 2006 22:12 UTC](https://rubytalk.org/t/reading-website/30080/10 "2006-08-17T22:12:11Z")
> Aha! Yes it was something old. > It opened the file with append.. > > And, sorry for all the questions.. (im really trying to learn) 🙂 > But is there any thing like this in ruby? > > --VB code-- > > If InStr(1, Text1.Text, "String to find", 1) \> 0 Then Msgbox "I found > the string!" > > -------- > > ( If InStr(Start,String1,String2,CompareMode)\>0 Then Msgbox "I found > the string!") Does VB index strings starting with 1? For very simple seaching, use String#include?   \>\> "foobar".include? "ooba"   =\> true If you want to check if there's a substring to be found -after- (and not including) the second character, for example, then I'd use a substring:   \>\> "foobar"[2..-1].include? "ooba"   =\> false   \>\> "foobar"[2..-1].include? "obar"   =\> true What VB's CompareMode is, I have no idea. Case sensitivity? If so, I'd personally manually use String#downcase in the conditional. And, of course, there's always regular expressions, the swiss army knife of string searching, but those are for a longer discussion and you can probably find material for those out there that explains them rather nicely. David Vallner > **···** > > On Thu, 17 Aug 2006 23:55:05 +0200, fabsy \ wrote: ---
**Author:** ![fabsy](https://avatars.discourse-cdn.com/v4/letter/f/5daacb/32.png) [@fabsy](https://rubytalk.org/u/fabsy)\ **Post date:** [17 August 2006 22:40 UTC](https://rubytalk.org/t/reading-website/30080/11 "2006-08-17T22:40:11Z")
David Vallner skrev: > \> Aha! Yes it was something old. > \> It opened the file with append.. > \> > \> And, sorry for all the questions.. (im really trying to learn) 🙂 > \> But is there any thing like this in ruby? > \> > \> --VB code-- > \> > \> If InStr(1, Text1.Text, "String to find", 1) \> 0 Then Msgbox "I found > \> the string!" > \> > \> -------- > \> > \> ( If InStr(Start,String1,String2,CompareMode)\>0 Then Msgbox "I found > \> the string!") > \> > \> > > Does VB index strings starting with 1? > > For very simple seaching, use String#include? > >   \>\> "foobar".include? "ooba" >   =\> true > > If you want to check if there's a substring to be found -after- (and not > including) the second character, for example, then I'd use a substring: > >   \>\> "foobar"[2..-1].include? "ooba" >   =\> false > >   \>\> "foobar"[2..-1].include? "obar" >   =\> true > > What VB's CompareMode is, I have no idea. Case sensitivity? If so, I'd > personally manually use String#downcase in the conditional. > > And, of course, there's always regular expressions, the swiss army knife > of string searching, but those are for a longer discussion and you can > probably find material for those out there that explains them rather > nicely. > > David Vallner Ok! But that doesn't tell me the position of the string.. Say that you have a textfile with the sentence "Could you pass me the milk please?" and I make a search for the string "milk". Then I want to know the position of where the search word starts so I can extract the word into a variable.... How do I do that? And yes VB index strings start with 1.. > **···** > > > On Thu, 17 Aug 2006 23:55:05 +0200, fabsy \ wrote: ---
**Author:** ![David\_Vallner](https://avatars.discourse-cdn.com/v4/letter/d/eb9ed0/32.png) [@David\_Vallner](https://rubytalk.org/u/David_Vallner)\ **Post date:** [17 August 2006 23:00 UTC](https://rubytalk.org/t/reading-website/30080/12 "2006-08-17T23:00:27Z")
> Ok! > But that doesn't tell me the position of the string.. > Say that you have a textfile with the sentence "Could you pass me the > milk please?" and I make a search for the string "milk". Then I want to > know the position of where the search word starts so I can extract the > word into a variable.... How do I do that? Use String#index, or String#=~. The former works with strings and regular expressions, the latter requires regular expressions, but I think it also saves the match data into the special global variable I don't use or remember the name of... Anyhoo:   \>\> "foobar".index "ooba"   =\> 1   \>\> "foobar".index "bar"   =\> 3   \>\> "foobar".index /bar/   =\> 3   \>\> "foobar".index /ooba/   =\> 1 For word extraction, I'm usually very, very bad doing those with substring indexes and abuse regexps wildly each and every time I need to get some piece of text out of another one. David Vallner > **···** > > On Fri, 18 Aug 2006 00:40:11 +0200, fabsy \ wrote: