# Extracting Data from a Webpage **URL:** \ **Category:** ruby-talk\ **Created:** [27 January 2008 03:21 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059 "2008-01-27T03:21:47Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![12345678\_123456789](https://avatars.discourse-cdn.com/v4/letter/1/3ec8ea/32.png) [@12345678\_123456789](https://rubytalk.org/u/12345678_123456789)\ **Post date:** [27 January 2008 03:21 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/1 "2008-01-27T03:21:47Z")
Hello everyone. I was wondering if anyone knew a way to extract the web page title off of a specific URL that you input into a program? I give it the URL, say [www.google.com](http://www.google.com). It then gives me "Google" - it's title. Then also, is there anyway that the program could extract the next 5 characters - after a certain phrase that doesn't change on the webpage? Thanks for your help in advance! > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Steve\_Ross](https://avatars.discourse-cdn.com/v4/letter/s/3d9bf3/32.png) [@Steve\_Ross](https://rubytalk.org/u/Steve_Ross)\ **Post date:** [27 January 2008 04:08 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/2 "2008-01-27T04:08:19Z")
[http://code.whytheluckystiff.net/hpricot/](http://code.whytheluckystiff.net/hpricot/) It's a snap. > **···** > > On Jan 26, 2008, at 7:21 PM, Tj Superfly wrote: > > > Hello everyone. > > > > I was wondering if anyone knew a way to extract the web page title off > > of a specific URL that you input into a program? > > > > I give it the URL, say [www.google.com](http://www.google.com). It then gives me "Google" - it's > > title. > > > > Then also, is there anyway that the program could extract the next 5 > > characters - after a certain phrase that doesn't change on the webpage? > > > > Thanks for your help in advance! > > -- > > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [27 January 2008 04:42 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/3 "2008-01-27T04:42:18Z")
Tj Superfly wrote: > Hello everyone. > > I was wondering if anyone knew a way to extract the web page title off > of a specific URL that you input into a program? > > I give it the URL, say [www.google.com](http://www.google.com). It then gives me "Google" - it's > title. > > Then also, is there anyway that the program could extract the next 5 > characters - after a certain phrase that doesn't change on the webpage? > > Thanks for your help in advance! You can do something like this: require 'open-uri' url = "[http://www.google.com](http://www.google.com)" open(url) do |f|   f.each do |line|     if md\_obj = /\(.\*)\<\/title\>/.match(line)       puts md\_obj[1]     end     if md\_obj = /type=(.{6})/.match(line)       puts md\_obj[1]     end   end end Ruby also has various html parsing libraries that allow you to search html documents by tag name, tag position, etc. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![W\_James](https://avatars.discourse-cdn.com/v4/letter/w/e274bd/32.png) [@W\_James](https://rubytalk.org/u/W_James)\ **Post date:** [27 January 2008 07:50 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/4 "2008-01-27T07:50:00Z")
"www.google.com"[/(www.)?(.\*)./,2].capitalize     ==\>"Google" "google.com"[/(www\.)?(.\*)\./,2].capitalize     ==\>"Google" > **···** > > On Jan 26, 9:21 pm, Tj Superfly \ wrote: > > > Hello everyone. > > > > I was wondering if anyone knew a way to extract the web page title off > > of a specific URL that you input into a program? > > > > I give it the URL, say [www.google.com](http://www.google.com). It then gives me "Google" - its > > title. ---
**Author:** ![W\_James](https://avatars.discourse-cdn.com/v4/letter/w/e274bd/32.png) [@W\_James](https://rubytalk.org/u/W_James)\ **Post date:** [28 January 2008 01:59 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/5 "2008-01-28T01:59:57Z")
> I was wondering if anyone knew a way to extract the web page title off > of a specific URL that you input into a program? require 'net/http' puts Net::HTTP.new('www.google.com').get('/').   body[/\(.\*?)\<.title\>/i,1] ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [27 January 2008 05:21 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/6 "2008-01-27T05:21:06Z")
7stud -- wrote: > You can do something like this: > > require 'open-uri' > > url = "[http://www.google.com](http://www.google.com)" > > open(url) do |f| >   f.each do |line| >     if md\_obj = /\(.\*)\<\/title\>/.match(line) >       puts md\_obj[1] >     end > >     if md\_obj = /type=(.{6})/.match(line) >       puts md\_obj[1] >     end > >   end > end This should be more efficient: require 'open-uri' url = "[http://www.google.com](http://www.google.com)" title\_re = Regexp.new(/\(.\*)\<\/title\>/) text\_re = Regexp.new(/type=(.{5})/) open(url) do |f|   f.each do |line|     if md\_obj = title\_re.match(line)       puts md\_obj[1]     end     if md\_obj = text\_re.match(line)       puts md\_obj[1]       break     end   end end --output: Google hidde #first 5 chars of 'hidden' > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [28 January 2008 04:24 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/7 "2008-01-28T04:24:20Z")
William James wrote: > > I was wondering if anyone knew a way to extract the web page title off > > of a specific URL that you input into a program? > > require 'net/http' > puts Net::HTTP.new('www.google.com').get('/'). >   body[/\(.\*?)\<.title\>/i,1] Nice. I tested your code and it works for me. But my reading of the docs says that it shouldn't work: new() doesn't open a connection, and get(), "Gets data from path on the connected-to host." The docs seem to want you to do something like: resp\_obj = Net::HTTP.get\_response('[http://www.google.com](http://www.google.com)', '/index.html') page = resp\_obj.body > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![12345678\_123456789](https://avatars.discourse-cdn.com/v4/letter/1/3ec8ea/32.png) [@12345678\_123456789](https://rubytalk.org/u/12345678_123456789)\ **Post date:** [27 January 2008 16:34 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/8 "2008-01-27T16:34:48Z")
> This should be more efficient: > > require 'open-uri' > > url = "[http://www.google.com](http://www.google.com)" > title\_re = Regexp.new(/\(.\*)\<\/title\>/) > text\_re = Regexp.new(/type=(.{5})/) > > open(url) do |f| >   f.each do |line| >     if md\_obj = title\_re.match(line) >       puts md\_obj[1] >     end > >     if md\_obj = text\_re.match(line) >       puts md\_obj[1] >       break >     end > >   end > end > > --output: > Google > hidde #first 5 chars of 'hidden' I receive this eror message when trying this code. DENTIFIER, expecting $end endndndreakmd\_obj[1]\_re.match(line))/title\>/) Any suggestions? I did try the other clip of code posted here, but got more errors than this one. =/ I'm reading up on that link posted in the 2nd post to see if I can figure any of this out. Thanks. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [28 January 2008 01:39 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/9 "2008-01-28T01:39:39Z")
7stud -- wrote: > title\_re = Regexp.new(/\(.\*)\<\/title\>/) While that regex works for [www.google.com](http://www.google.com), in order for the regex to be more general, the regex should be: title\_re = Regexp.new(/\(.\*)\<\/title\>/m) and then to output the match: puts md\_obj[1].strip() > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![JoeP](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@JoeP](https://rubytalk.org/u/JoeP)\ **Post date:** [28 January 2008 04:42 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/10 "2008-01-28T04:42:11Z")
7stud -- wrote: > Nice. I tested your code and it works for me. But my reading of the > docs says that it shouldn't work: new() doesn't open a connection, and > get(), "Gets data from path on the connected-to host." The docs seem > to want you to do something like: > > resp\_obj = Net::HTTP.get\_response('[http://www.google.com](http://www.google.com)', > '/index.html') > page = resp\_obj.body Reading the Net::HTTP docs here: [http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html#M001127](http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html#M001127) It says that Net::HTTP#get will: "Send a GET request to the target and return the response as a string" and Net::HTTP#get\_response will: "Send a GET request to the target and return the response as a Net::HTTPResponse object" The #new in this case is optional because both methods are class methods or instance methods? Someone might be able to clarify this a part a little more. But the examples at that doc url don't even use New::HTTP#new. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [27 January 2008 20:24 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/11 "2008-01-27T20:24:55Z")
Tj Superfly wrote: > I receive this eror message when trying this code. > > DENTIFIER, expecting $end > endndndreakmd\_obj[1]\_re.match(line))/title\>/) > > Any suggestions? 1) Learn some basic ruby? 2) Learn how to post a question on a computer programming forum? > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [28 January 2008 09:58 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/12 "2008-01-28T09:58:49Z")
Joseph Pecoraro wrote: > 7stud -- wrote: > > > Nice. I tested your code and it works for me. But my reading of the > > docs says that it shouldn't work: new() doesn't open a connection, and > > get(), "Gets data from path on the connected-to host." The docs seem > > to want you to do something like: > > > > resp\_obj = Net::HTTP.get\_response('[http://www.google.com](http://www.google.com)', > > '/index.html') > > page = resp\_obj.body > > Reading the Net::HTTP docs here: > [http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html#M001127](http://ruby-doc.org/stdlib/libdoc/net/http/rdoc/classes/Net/HTTP.html#M001127) > > It says that Net::HTTP#get will: > "Send a GET request to the target and return the response as a string" > > and Net::HTTP#get\_response will: > "Send a GET request to the target and return the response as a > Net::HTTPResponse object" > > The #new in this case is optional because both methods are class methods > or instance methods? According to the docs, Net::HTTP has class methods: get() get\_response() and an instance method: get() As with all ruby classes, new() creates an instance. Therefore, in the code example I was wondering about: > puts Net::HTTP.new('www.google.com').get('/'). >   body[/\(.\*?)\<.title\>/i,1] new() creates an instance, which is being used to call get(), so the version of get() being called is the instance method. Yet, the docs say the get() instance method "Gets data from path on the connected-to host". What connected to host? According to the docs on new() it says, "This method does not open the TCP connection." In addition, the get() version in that code cannot be the class method version because the class method version returns a String and Strings do not have a body() method, which is the next method call. > **···** > > > Someone might be able to clarify this a part a > > little more. But the examples at that doc url don't even use > > New::HTTP#new. > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![12345678\_123456789](https://avatars.discourse-cdn.com/v4/letter/1/3ec8ea/32.png) [@12345678\_123456789](https://rubytalk.org/u/12345678_123456789)\ **Post date:** [27 January 2008 21:57 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/13 "2008-01-27T21:57:19Z")
7stud -- wrote: > Tj Superfly wrote: > > > I receive this eror message when trying this code. > > > > DENTIFIER, expecting $end > > endndndreakmd\_obj[1]\_re.match(line))/title\>/) > > > > Any suggestions? > > 1) Learn some basic ruby? > > 2) Learn how to post a question on a computer programming forum? Anyone else know what the matter is? > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [28 January 2008 10:08 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/14 "2008-01-28T10:08:06Z")
7stud -- wrote: > As with all ruby classes, new() creates an instance. Therefore, in the > code example I was wondering about: > > > puts Net::HTTP.new('www.google.com').get('/'). > >   body[/\(.\*?)\<.title\>/i,1] > > new() creates an instance, which is being used to call get(), so the > version of get() being called is the instance method. Yet, the docs say > the get() instance method "Gets data from path on the connected-to > host". What connected to host? According to the docs on new() it says, > "This method does not open the TCP connection." As far as I can tell, you should have to call start() on a Net::HTTP instance in order to open a connection, e.g.: str = Net::HTTP.new('www.google.com').start().get('/').body > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [27 January 2008 22:26 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/15 "2008-01-27T22:26:09Z")
Tj Superfly wrote: > 7stud -- wrote: > > > Tj Superfly wrote: > > > > > I receive this eror message when trying this code. > > > > > > DENTIFIER, expecting $end > > > endndndreakmd\_obj[1]\_re.match(line))/title\>/) > > > > > > Any suggestions? > > > > 1) Learn some basic ruby? > > > > 2) Learn how to post a question on a computer programming forum? > > Anyone else know what the matter is? How to post a question on a computer programming Forum: 1) Post a simple example program that demonstrates your problem. 2) Post the error message in its entirety--not an unintelligible portion of it. 3) Post your question about the code. 4) Use a descriptive title for your post-- not something like "URGENT...HELP ME!" 5) Proof read and spell check your post before clicking submit. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![fedzor](https://avatars.discourse-cdn.com/v4/letter/f/bbe5ce/32.png) [@fedzor](https://rubytalk.org/u/fedzor)\ **Post date:** [27 January 2008 22:49 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/16 "2008-01-27T22:49:21Z")
I believe that $end means you're missing some sort of end delimiter, but NOT 'end'. Check for {} or / / for regexp Also, if you can, have your editor do an autoformat thing so you can see where the indentation screws up. > **···** > > On Jan 27, 2008, at 4:57 PM, Tj Superfly wrote: > > > 7stud -- wrote: > > > > > Tj Superfly wrote: > > > > > > > I receive this eror message when trying this code. > > > > > > > > DENTIFIER, expecting $end > > > > endndndreakmd\_obj[1]\_re.match(line))/title\>/) > > > > > > > > Any suggestions? ---
**Author:** ![Marc\_Heiler](https://avatars.discourse-cdn.com/v4/letter/m/e8c25b/32.png) [@Marc\_Heiler](https://rubytalk.org/u/Marc_Heiler)\ **Post date:** [27 January 2008 23:16 UTC](https://rubytalk.org/t/extracting-data-from-a-webpage/44059/17 "2008-01-27T23:16:42Z")
> [http://code.whytheluckystiff.net/hpricot/](http://code.whytheluckystiff.net/hpricot/) > It's a snap. I believe hpricot, as fine as it may be, is a little bit overkill for such a task. At best a simple task should remain simple, at least as simple as possible. > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).