# Bounty: $250 - Successfully log in to AOL Webmail with Mechanize **URL:** \ **Category:** ruby-talk\ **Created:** [30 October 2007 19:58 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802 "2007-10-30T19:58:17Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Dave\_Myron](https://avatars.discourse-cdn.com/v4/letter/d/db5fbb/32.png) [@Dave\_Myron](https://rubytalk.org/u/Dave_Myron)\ **Post date:** [30 October 2007 19:58 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/1 "2007-10-30T19:58:17Z")
We're about to release a gem called Blackbook that will automate scraping contacts from various services. We have Gmail, Hotmail and YahooMail all done - they were easy - but AOL Webmail (http:// [webmail.aol.com](http://webmail.aol.com)) has stumped us. After countless hours comparing LiveHTTPHeader captures of successful logins using Firefox (with meta redirects and javascript disabled) with logs of unsuccessful sessions using Mechanize, we're finally putting a bounty on AOL's head. Comparing the Firefox session with Mechanize's shows they're nearly identical (seems like all but the Accepts header matches) until the very last request: Firefox is authenticated and Mechanize is diverted to an error page. We're offering a $250 bounty to the first person to successfully log in to AOL Webmail with Mechanize. If you'd like a starting point (which might not help much since it ends up at an error page), here's our code: > **···** > > ============= > > require 'rubygems' > require 'mechanize' > > agent = WWW::Mechanize.new > agent.user\_agent\_alias = 'Windows IE 6' > > page = agent.get( '[http://webmail.aol.com](http://webmail.aol.com)' ) > > page = agent.get( page.body.scan(/snsRedir\("(https:[^"] > +)/).first.first ) > > form = [page.forms.name](http://page.forms.name)('AOLLoginForm').first > form.loginId = '\' > form.password = '\' > page = agent.submit(form, form.buttons.first) > > page = agent.get( page.body.scan(/onLoad="checkError[^\)] > +/).first.scan(/'([^']+)'/).last.first ) > > ============= > > That mimics what we can do in Firefox (cutting and pasting where we > use #scan in the code) and the last request should authenticate us > (with several cookies including one named Auth) but we get redirected > to an error page. > > Good luck. In addition to your bounty, your work will also net the > community another gem. > > Dave ---
**Author:** ![Shanti\_Braford](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Shanti\_Braford](https://rubytalk.org/u/Shanti_Braford)\ **Post date:** [31 October 2007 01:11 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/2 "2007-10-31T01:11:12Z")
Hi Dave, I don't think I can help you re: Mechanize and AOL Webmail (at the moment at least). Just wanted to make sure you've seen this: [http://rubyforge.org/projects/contacts/](http://rubyforge.org/projects/contacts/) ... in case any pooling of efforts would help. Looking forward to seeing your results! Cheers, - Shanti > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Peter\_Szinek3](https://avatars.discourse-cdn.com/v4/letter/p/ea5d25/32.png) [@Peter\_Szinek3](https://rubytalk.org/u/Peter_Szinek3)\ **Post date:** [31 October 2007 08:35 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/3 "2007-10-31T08:35:25Z")
Do you \*absolutely\* need mechanize? Cheers, Peter > **···** > > dave.myron@contentfree.com wrote: > > > We're about to release a gem called Blackbook that will automate > > scraping contacts from various services. We have Gmail, Hotmail and > > YahooMail all done - they were easy - but AOL Webmail (http:// > > webmail.aol.com) has stumped us. > > \_\_\_ > [http://www.rubyrailways.com](http://www.rubyrailways.com) > [http://scrubyt.org](http://scrubyt.org) ---
**Author:** ![mortee](https://avatars.discourse-cdn.com/v4/letter/m/50afbb/32.png) [@mortee](https://rubytalk.org/u/mortee)\ **Post date:** [1 November 2007 02:24 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/4 "2007-11-01T02:24:13Z")
I think I have it figured it out. I have a script that can log you into AOL webmail using Mechanize, to the point where it greets you with "Hi, \". If that's what you needed, then I have the solution for you. mortee > **···** > > dave.myron@contentfree.com wrote: > > > We're about to release a gem called Blackbook that will automate > > scraping contacts from various services. We have Gmail, Hotmail and > > YahooMail all done - they were easy - but AOL Webmail (http:// > > webmail.aol.com) has stumped us. > > > > After countless hours comparing LiveHTTPHeader captures of successful > > logins using Firefox (with meta redirects and javascript disabled) > > with logs of unsuccessful sessions using Mechanize, we're finally > > putting a bounty on AOL's head. > > > > Comparing the Firefox session with Mechanize's shows they're nearly > > identical (seems like all but the Accepts header matches) until the > > very last request: Firefox is authenticated and Mechanize is diverted > > to an error page. > > > > We're offering a $250 bounty to the first person to successfully log > > in to AOL Webmail with Mechanize. > > > > If you'd like a starting point (which might not help much since it > > ends up at an error page), here's our code: > > > > ============= > > > > require 'rubygems' > > require 'mechanize' > > > > agent = WWW::Mechanize.new > > agent.user\_agent\_alias = 'Windows IE 6' > > > > page = agent.get( '[http://webmail.aol.com](http://webmail.aol.com)' ) > > > > page = agent.get( page.body.scan(/snsRedir\("(https:[^"] > > +)/).first.first ) > > > > form = page.forms.name('AOLLoginForm').first > > form.loginId = '\' > > form.password = '\' > > page = agent.submit(form, form.buttons.first) > > > > page = agent.get( page.body.scan(/onLoad="checkError[^\)] > > +/).first.scan(/'([^']+)'/).last.first ) > > > > ============= > > > > That mimics what we can do in Firefox (cutting and pasting where we > > use #scan in the code) and the last request should authenticate us > > (with several cookies including one named Auth) but we get redirected > > to an error page. > > > > Good luck. In addition to your bounty, your work will also net the > > community another gem. > > > > Dave ---
**Author:** ![Dave\_Myron](https://avatars.discourse-cdn.com/v4/letter/d/db5fbb/32.png) [@Dave\_Myron](https://rubytalk.org/u/Dave_Myron)\ **Post date:** [31 October 2007 18:56 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/5 "2007-10-31T18:56:32Z")
> Just wanted to make sure you've seen this:[http://rubyforge.org/projects/contacts/](http://rubyforge.org/projects/contacts/) We hadn't - and looking now I'm not sure how we missed it. We're doing things a little different (we plan on having different exporters too like csv, xml, etc) so I'm not sure how much merging could happen (but I'll check in with them). Unfortunately, they don't have AOL figured out either. It's weird that these PHP scripts (using curl) seem to be able to log in fine but we can't (and it seems MOG couldn't either). ---
**Author:** ![Dave\_Myron](https://avatars.discourse-cdn.com/v4/letter/d/db5fbb/32.png) [@Dave\_Myron](https://rubytalk.org/u/Dave_Myron)\ **Post date:** [31 October 2007 19:28 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/6 "2007-10-31T19:28:15Z")
> Do you \*absolutely\* need mechanize? It's helpful and the other completed importers (Gmail, Yahoo, Hotmail) use it already. To keep dependencies down I'd \*like\* it to use Mechanize, but if you've got something using scRUBYt then I'm sure it could be translated (or at least help highlight where Mechanize is falling down). Dave ---
**Author:** ![Dave\_Myron](https://avatars.discourse-cdn.com/v4/letter/d/db5fbb/32.png) [@Dave\_Myron](https://rubytalk.org/u/Dave_Myron)\ **Post date:** [1 November 2007 05:30 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/7 "2007-11-01T05:30:27Z")
> AOL webmail using Mechanize, to the point where it greets you with "Hi, > \". If that's what you needed, then I have the solution for you. > > mortee If you want to send it directly to me, that's fine. Also feel free to paste it here. Dave ---
**Author:** ![Konrad\_Meyer](https://avatars.discourse-cdn.com/v4/letter/k/df705f/32.png) [@Konrad\_Meyer](https://rubytalk.org/u/Konrad_Meyer)\ **Post date:** [31 October 2007 23:39 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/8 "2007-10-31T23:39:45Z")
Quoth dave.myron@contentfree.com: > \> Just wanted to make sure you've seen this:[http://rubyforge.org/projects/contacts/](http://rubyforge.org/projects/contacts/) > We hadn't - and looking now I'm not sure how we missed it. We're doing > things a little different (we plan on having different exporters too > like csv, xml, etc) so I'm not sure how much merging could happen (but > I'll check in with them). Unfortunately, they don't have AOL figured > out either. > > It's weird that these PHP scripts (using curl) seem to be able to log > in fine but we can't (and it seems MOG couldn't either). Could AOL be rejecting you based on UserAgent? > **···** > > -- > Konrad Meyer \ [http://konrad.sobertillnoon.com/](http://konrad.sobertillnoon.com/) ---
**Author:** ![mortee](https://avatars.discourse-cdn.com/v4/letter/m/50afbb/32.png) [@mortee](https://rubytalk.org/u/mortee)\ **Post date:** [1 November 2007 19:12 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/9 "2007-11-01T19:12:45Z")
I've sent a private mail to you; have you received it?... Sorry for the noise. mortee > **···** > > dave.myron@contentfree.com wrote: > > > > AOL webmail using Mechanize, to the point where it greets you with "Hi, > > > \". If that's what you needed, then I have the solution for you. > > > > > > mortee > > > > If you want to send it directly to me, that's fine. Also feel free to > > paste it here. ---
**Author:** ![yermej](https://avatars.discourse-cdn.com/v4/letter/y/b2d939/32.png) [@yermej](https://rubytalk.org/u/yermej)\ **Post date:** [1 November 2007 01:55 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/10 "2007-11-01T01:55:02Z")
This line agent.user\_agent\_alias = 'Windows IE 6' in the sample code results in a UserAgent that matches IE 6. > **···** > > On Oct 31, 6:39 pm, Konrad Meyer \ wrote: > > > Quoth dave.my...@contentfree.com:\> \> Just wanted to make sure you've seen > > > > this:[http://rubyforge.org/projects/contacts/](http://rubyforge.org/projects/contacts/) > > > > \> We hadn't - and looking now I'm not sure how we missed it. We're doing > > \> things a little different (we plan on having different exporters too > > \> like csv, xml, etc) so I'm not sure how much merging could happen (but > > \> I'll check in with them). Unfortunately, they don't have AOL figured > > \> out either. > > > > \> It's weird that these PHP scripts (using curl) seem to be able to log > > \> in fine but we can't (and it seems MOG couldn't either). > > > > Could AOL be rejecting you based on UserAgent? > > > > -- > > Konrad Meyer \[http://konrad.sobertillnoon.com/](http://konrad.sobertillnoon.com/) > > > > signature.asc > > 1KDownload ---
**Author:** ![Dave\_Myron](https://avatars.discourse-cdn.com/v4/letter/d/db5fbb/32.png) [@Dave\_Myron](https://rubytalk.org/u/Dave_Myron)\ **Post date:** [2 November 2007 00:57 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/11 "2007-11-02T00:57:19Z")
You're the man, Marton! Consider the bounty won! Marton found a bug in Mechanize's #to\_absolute\_uri method caused by the use of #zip which caused already-escaped characters beyond the first to be dropped. He replaced it with some SyncEnumerator hackery. I'll let the Aaron Patterson know about the fix to Mechanize. Good job Marton. We (and Blackbook and probably Mechanize, too) appreciate your genius (and tolerance level for reading through that nasty bit of Javascript to highlight the problem). Dave ---
**Author:** ![mortee](https://avatars.discourse-cdn.com/v4/letter/m/50afbb/32.png) [@mortee](https://rubytalk.org/u/mortee)\ **Post date:** [4 November 2007 08:25 UTC](https://rubytalk.org/t/bounty-250-successfully-log-in-to-aol-webmail-with-mechanize/41802/12 "2007-11-04T08:25:37Z")
> You're the man, Marton! Consider the bounty won! My pleasure (: > Marton found a bug in Mechanize's #to\_absolute\_uri method caused by > the use of #zip which caused already-escaped characters beyond the > first to be dropped. He replaced it with some SyncEnumerator hackery. > I'll let the Aaron Patterson know about the fix to Mechanize. Something like that. Thanks if you let him know, then I won't have to. > Good job Marton. We (and Blackbook and probably Mechanize, too) > appreciate your genius (and tolerance level for reading through that > nasty bit of Javascript to highlight the problem). It was a nice task. Let me point out here that Dave and his company are really generous, we got along pretty nicely. regards mortee > **···** > > dave.myron@contentfree.com wrote: