Hey there everyone. I'm having a slight problem using Mechanize. I'm
trying to scrape the yellowpages.com, and extract information about each
business listing. I'm extracting all the information I want, except for
one small portion: the business's website. It is the href inside of a
link that I am trying to scrape. As far as I know, I'm following the
correct xpath rules, but I can't seem to get the part I want. One
tricky thing that I've had to deal with is that not every listing has a
website. The website link and the "learn more" link are very similar,
xpath-wise, so I have to use an if statement to check the inner text of
both of them to make sure that I'm extracting the xpath one.
Jethrow, thanks but that's not quite what I need. I need to extract
this link's href attribute, which is the website of the buisness. I'm
using xpath, using the ".../a/@href" method which I believe is the
correct one. But it just doesn't extract anything! Any other ideas?
Hey there everyone. I'm having a slight problem using Mechanize. I'm
trying to scrape the yellowpages.com, and extract information about each
business listing. I'm extracting all the information I want, except for
one small portion: the business's website. It is the href inside of a
link that I am trying to scrape. As far as I know, I'm following the
correct xpath rules, but I can't seem to get the part I want. One
tricky thing that I've had to deal with is that not every listing has a
website. The website link and the "learn more" link are very similar,
xpath-wise, so I have to use an if statement to check the inner text of
both of them to make sure that I'm extracting the xpath one.
a leading "/" means that you want the xpath search to begin from the
root of the document. "./" means to start from the context node, in
this case `website`.
···
On Sat, Oct 16, 2010 at 3:50 PM, Corey Watts <cwatts@westmont.edu> wrote:
Thanks Mike. I've updated the code. Putting "./" in my code messed up
what the code grabbed, so I eliminated all leading "/" and it worked.
However, the website part still doesn't work. It looks like it is
grabbing what is inside of the a tags, instead of grabbing the href
address. A typical listing output looks like:
McDonald's
1213 State St # B,
Santa Barbara
CA
93101
(805) 962-6976
<span class="raquo">»</span>
?
Website
["Restaurants", "Fast Food Restaurants", "American Restaurants"]
The three lines: <span>, ?, and Website are all grabbed from the website
= website.search(...) line, but it's grabbing the wrong thing! Do you
have any more suggestions?
I still haven't figured this out. Perhaps I should phrase the question
a different way...
What is the preferred method of extracting the href attribute from a
link? I've tried doing it using .search() and searching for the xml @href attribute. For some reason that's not working for me.
Is there a different way of extracting this attribute, without using
.search and an xml path? I'm sure mechanize has some other method
too...
Thanks! I eventually came up with something like this:
website =
page.search("//div[@class='listing_content']/ul[@class='features']/li[1]")
# initializes the link variable with something
link = "super"
# The segment below grabs the URL of the listing's website, if it has
one. Otherwise it
# puts 'nil' into the link variable
if website.inner_text() == "» Website"
website.search("a").map do |l|
link = l['href']
end
else
link = "nil"
end
Robert, thank you so much for your help. I'm just starting out in ruby,
so I have a lot to learn! I've attached my code with the changes you
recommended.
Thanks again for your help. This script above is actually part of a
larger script, but I was just showing you the smaller portion for
clarity. I've attached the whole script, in order to put the for loop
in context. I'm using the for loop to grab multiple pages in
succession. Hopefully this makes it more clear.
I've made all the changes you recommended, except consolidating the puts
statements. Could you explain how I would do that? Thank you!
With this and a local version of the page I was able to get the info you want:
#!/bin/env ruby19
require 'nokogiri'
raw = File.read("restaurants.html", mode: "r:UTF-8")
puts raw.encoding
# raw.force_encoding 'UTF-8'
doc = Nokogiri.parse raw
doc.xpath('//div[@class="listing_content"]').each do |listing|
puts '----------------------------------------'
# p listing.to_s[0...10]+"..."
puts listing
puts '----------------------------------------'
# p listing.xpath('.//a//text()').map(&:to_s)
listing.xpath('.//a[@href and contains(text(),"Website")]/@href').each do |a|
p a.value
end
puts
end
Cheers
robert
···
On Wed, Oct 27, 2010 at 2:57 AM, Corey Watts <cwatts@westmont.edu> wrote:
I still haven't figured this out. Perhaps I should phrase the question
a different way...
What is the preferred method of extracting the href attribute from a
link? I've tried doing it using .search() and searching for the xml @href attribute. For some reason that's not working for me.
Is there a different way of extracting this attribute, without using
.search and an xml path? I'm sure mechanize has some other method
too...
Wouldn't it be better to first search for all div with
class=listing_content and then extract data for that single listing
from there?
# initializes the link variable with something
link = "super"
"link = nil" is sufficient.
# The segment below grabs the URL of the listing's website, if it has
one. Otherwise it
# puts 'nil' into the link variable
if website.inner_text() == "» Website"
You can use my XPath to find those links.
website.search("a").map do |l|
#map is nonsense here - you only need #each.
link = l['href']
end
else
link = "nil"
Didn't you mean "link = nil"?
Kind regards
robert
···
On Thu, Oct 28, 2010 at 8:54 PM, Corey Watts <cwatts@westmont.edu> wrote:
for x in 1...2 will execute the loop body exactly once. Are you sure
this is what you want?
You can combine all the puts at the end in a single statement.
You can replace all the ".gsub(/^\s+/, "").gsub(/\s+$/, $/)" by
".strip. You do not need to append a newline because puts does that
already.
Why do you use 'entry.search("a")' and do not include the "a" in the
path expression?
Cheers
robert
···
On Fri, Oct 29, 2010 at 9:15 AM, Corey Watts <cwatts@westmont.edu> wrote:
Robert, thank you so much for your help. I'm just starting out in ruby,
so I have a lot to learn! I've attached my code with the changes you
recommended.