Alexander York wrote:
Troy Denkinger wrote:
Where I have interest in going with Ruby
-Screen scraping eBay for specific products/information
-(unrelated but need help finding this)Image recognition software
-Web based management for screen scraping and storing of data/pdf
creation of auctions
Screen scraping is good fun and all, but have you looked into the web
services APIs that eBay makes available? There may be much easier ways
get at data than screen scraping. Look at the "Developers" link at the
bottom of their homepage.
This is unquestionably true: no web scraper will ever (?) give you 100%
reliability and robustness and ... that a WS API can offer. If your thing
is mission critical, and the API offers everything you need, no one is
questioning that it is the way to go - even if the other alternative is
Hmm looks like good advice, could you perhaps give me an example of how
I could apply this with ruby? (I'm still new)
Do you have the Ruby Cookbook? There is one recipe for using the ebay API...
The scraping i will need
to do is going to be pretty advanced I think
The present version of scRUBYt! (about 0.0.8) works like this:
ebay_data = Scrubyt::Extractor define do
#Navigate to our page
fill_textfield 'dell laptop'
#Construct the scraper
name "DELL LATITUDE C600 P3 1.0 LAPTOP NOTEBOOK 256MB 20GB HD"
next_page "8 9 Next", :limit => 5
So, this extractor does the following: first navigates to your page of
interest automatically as you would do it (powered by Mechanize - from
then later on Selenium), Then the scraper is constructed.
As you can see, there is no need for any technical knowledge (powered by
Hpricot); You just show an example record and all the others are
extracted automatically. Moreover, only those records are extracted
which have a price (i.e. not the 'buy now' ones). Another goodie for
free is the navigation to the next pages - in this case the first 5
pages (you can do also unlimited as well).
Now, this is just a part of the functionality even of the version which
is on my HDD - I am planning to add more stuff into 0.1.0 - and even
that will be just the top of the iceberg of the features I am planning
to add long term.
And of course the best thing is that it *works* - it's not a toy (at
least not on the pages I have tried it so far) - and though I am sure
there will be problems with the earlier versions, I count on the
community to help to catch these and propose features important for them
so I guess the future might be bright for scRUBYt!
On 12/21/06, Alexander York <email@example.com> wrote: