We're about to release a gem called Blackbook that will automate
scraping contacts from various services. We have Gmail, Hotmail and
YahooMail all done - they were easy - but AOL Webmail (http:// webmail.aol.com) has stumped us.
After countless hours comparing LiveHTTPHeader captures of successful
logins using Firefox (with meta redirects and javascript disabled)
with logs of unsuccessful sessions using Mechanize, we're finally
putting a bounty on AOL's head.
Comparing the Firefox session with Mechanize's shows they're nearly
identical (seems like all but the Accepts header matches) until the
very last request: Firefox is authenticated and Mechanize is diverted
to an error page.
We're offering a $250 bounty to the first person to successfully log
in to AOL Webmail with Mechanize.
If you'd like a starting point (which might not help much since it
ends up at an error page), here's our code:
That mimics what we can do in Firefox (cutting and pasting where we
use #scan in the code) and the last request should authenticate us
(with several cookies including one named Auth) but we get redirected
to an error page.
Good luck. In addition to your bounty, your work will also net the
community another gem.
We're about to release a gem called Blackbook that will automate
scraping contacts from various services. We have Gmail, Hotmail and
YahooMail all done - they were easy - but AOL Webmail (http://
webmail.aol.com) has stumped us.
I think I have it figured it out. I have a script that can log you into
AOL webmail using Mechanize, to the point where it greets you with "Hi,
<username>". If that's what you needed, then I have the solution for you.
mortee
···
dave.myron@contentfree.com wrote:
We're about to release a gem called Blackbook that will automate
scraping contacts from various services. We have Gmail, Hotmail and
YahooMail all done - they were easy - but AOL Webmail (http://
webmail.aol.com) has stumped us.
After countless hours comparing LiveHTTPHeader captures of successful
logins using Firefox (with meta redirects and javascript disabled)
with logs of unsuccessful sessions using Mechanize, we're finally
putting a bounty on AOL's head.
Comparing the Firefox session with Mechanize's shows they're nearly
identical (seems like all but the Accepts header matches) until the
very last request: Firefox is authenticated and Mechanize is diverted
to an error page.
We're offering a $250 bounty to the first person to successfully log
in to AOL Webmail with Mechanize.
If you'd like a starting point (which might not help much since it
ends up at an error page), here's our code:
That mimics what we can do in Firefox (cutting and pasting where we
use #scan in the code) and the last request should authenticate us
(with several cookies including one named Auth) but we get redirected
to an error page.
Good luck. In addition to your bounty, your work will also net the
community another gem.
We hadn't - and looking now I'm not sure how we missed it. We're doing
things a little different (we plan on having different exporters too
like csv, xml, etc) so I'm not sure how much merging could happen (but
I'll check in with them). Unfortunately, they don't have AOL figured
out either.
It's weird that these PHP scripts (using curl) seem to be able to log
in fine but we can't (and it seems MOG couldn't either).
It's helpful and the other completed importers (Gmail, Yahoo, Hotmail)
use it already. To keep dependencies down I'd *like* it to use
Mechanize, but if you've got something using scRUBYt then I'm sure it
could be translated (or at least help highlight where Mechanize is
falling down).
We hadn't - and looking now I'm not sure how we missed it. We're doing
things a little different (we plan on having different exporters too
like csv, xml, etc) so I'm not sure how much merging could happen (but
I'll check in with them). Unfortunately, they don't have AOL figured
out either.
It's weird that these PHP scripts (using curl) seem to be able to log
in fine but we can't (and it seems MOG couldn't either).
> We hadn't - and looking now I'm not sure how we missed it. We're doing
> things a little different (we plan on having different exporters too
> like csv, xml, etc) so I'm not sure how much merging could happen (but
> I'll check in with them). Unfortunately, they don't have AOL figured
> out either.
> It's weird that these PHP scripts (using curl) seem to be able to log
> in fine but we can't (and it seems MOG couldn't either).
Marton found a bug in Mechanize's #to_absolute_uri method caused by
the use of #zip which caused already-escaped characters beyond the
first to be dropped. He replaced it with some SyncEnumerator hackery.
I'll let the Aaron Patterson know about the fix to Mechanize.
Good job Marton. We (and Blackbook and probably Mechanize, too)
appreciate your genius (and tolerance level for reading through that
nasty bit of Javascript to highlight the problem).
Marton found a bug in Mechanize's #to_absolute_uri method caused by
the use of #zip which caused already-escaped characters beyond the
first to be dropped. He replaced it with some SyncEnumerator hackery.
I'll let the Aaron Patterson know about the fix to Mechanize.
Something like that. Thanks if you let him know, then I won't have to.
Good job Marton. We (and Blackbook and probably Mechanize, too)
appreciate your genius (and tolerance level for reading through that
nasty bit of Javascript to highlight the problem).
It was a nice task. Let me point out here that Dave and his company are
really generous, we got along pretty nicely.