# Scrape javascript content

**URL:** <https://rubytalk.org/t/scrape-javascript-content/58599>\
**Category:** ruby-talk\
**Created:** [20 May 2010 05:48 UTC](https://rubytalk.org/t/scrape-javascript-content/58599 "2010-05-20T05:48:05Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Phil\_Mcdonnell](https://avatars.discourse-cdn.com/v4/letter/p/e95f7d/32.png) [@Phil\_Mcdonnell](https://rubytalk.org/u/Phil_Mcdonnell)\
**Post date:** [20 May 2010 05:48 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/1 "2010-05-20T05:48:05Z")

</div>

I'm trying to scrape a page that hides some data behind a javascript  
function. Is there any way to get this data? I've been using  
Mechanize, but I'm not sure it can do this. Is there a better library  
to use for this type of thing?

The following is the interesting part of the page:

\<td class="colPlus" onclick="fireClick(this,0)"\>  
&nbsp;&nbsp;&nbsp;&nbsp;\<a id="iroc\_0" class="plus" href="#" onclick="return  
false;"\>&nbsp;\</a\>  
\</td\>

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![Josh\_Cheek](https://avatars.discourse-cdn.com/v4/letter/j/e79b87/32.png) [@Josh\_Cheek](https://rubytalk.org/u/Josh_Cheek)\
**Post date:** [20 May 2010 06:16 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/2 "2010-05-20T06:16:58Z")

</div>

You might check out Harmony:

[http://www.rubyinside.com/harmony-javascript-and-a-dom-environment-in-ruby-3001.html](http://www.rubyinside.com/harmony-javascript-and-a-dom-environment-in-ruby-3001.html)  
[http://rubygems.org/gems/harmony](http://rubygems.org/gems/harmony)

> **[GitHub - mynyml/harmony: Javascript + DOM in your ruby, the simple way](https://github.com/mynyml/harmony)**
>
> Javascript + DOM in your ruby, the simple way. Contribute to mynyml/harmony development by creating an account on GitHub.

> **···**
>
> On Thu, May 20, 2010 at 12:48 AM, Phil Mcdonnell \<phil.a.mcdonnell@gmail.com \> wrote:
> 
> > I'm trying to scrape a page that hides some data behind a javascript  
> > function. Is there any way to get this data? I've been using  
> > Mechanize, but I'm not sure it can do this. Is there a better library  
> > to use for this type of thing?
> > 
> > The following is the interesting part of the page:
> > 
> > \<td class="colPlus" onclick="fireClick(this,0)"\>  
> > &nbsp;&nbsp;&nbsp;\<a id="iroc\_0" class="plus" href="#" onclick="return  
> > false;"\>&nbsp;\</a\>  
> > \</td\>  
> > --  
> > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Brabuhr](https://avatars.discourse-cdn.com/v4/letter/b/919ad9/32.png) [@Brabuhr](https://rubytalk.org/u/Brabuhr)\
**Post date:** [20 May 2010 12:49 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/3 "2010-05-20T12:49:51Z")

</div>

> I'm trying to scrape a page that hides some data behind a javascript  
> function. Is there any way to get this data? I've been using  
> Mechanize, but I'm not sure it can do this. Is there a better library  
> to use for this type of thing?

[http://celerity.rubyforge.org/](http://celerity.rubyforge.org/)

> **[Watir Project](http://watir.com/)**
>
> Watir stands for Web Application Testing In Ruby. It facilitates the writing of automated tests by mimicking the behavior of a user interacting with a website.

> The following is the interesting part of the page:
> 
> \<td class="colPlus" onclick="fireClick(this,0)"\>  
> \<a id="iroc\_0" class="plus" href="#" onclick="return  
> false;"\>&nbsp;\</a\>  
> \</td\>

The \*really\* interesting part is what does the Javascript do 🙂 with  
(a potentially large) effort you may be able to "reverse-engineer" the  
javascript and emulate manually in mechanize. I.e. if the javascript  
builds a simple HTTP request, you may be able to send the same request  
from mechanize (possibly) without much effort.

> **···**
>
> On Thu, May 20, 2010 at 1:48 AM, Phil Mcdonnell \<phil.a.mcdonnell@gmail.com\> wrote:

---

<div class="post-metadata">

**Author:** ![Phil\_Mcdonnell](https://avatars.discourse-cdn.com/v4/letter/p/e95f7d/32.png) [@Phil\_Mcdonnell](https://rubytalk.org/u/Phil_Mcdonnell)\
**Post date:** [20 May 2010 16:14 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/4 "2010-05-20T16:14:09Z")

</div>

The other trick here is that this page is behind a login. Mechanize  
allows me to fill out the login form and holds onto the login  
credentials for me. Can harmony/celebrity/watir do this?

> The \*really\* interesting part is what does the Javascript do 🙂 with  
> (a potentially large) effort you may be able to "reverse-engineer" the  
> javascript and emulate manually in mechanize. I.e. if the javascript  
> builds a simple HTTP request, you may be able to send the same request  
> from mechanize (possibly) without much effort.

How would one do this? I'm somewhat new to javascript as I usually  
don't do front end engineering. I see the below definition of this  
function in the HTML page. Any way I can sniff out what it's actually  
doing? I'm looking to figure out what the fireClick method displays.

&nbsp;&nbsp;&nbsp;&nbsp;\<script type="text/javascript"\>  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var d = document.domain.split(".");  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;document.domain = d[d.length - 2] + "." + d[d.length - 1];  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var start = (new Date()).getTime();  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var fireClick = function(){};  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var omn\_hierarchy="US|AMEX|Ser|eStatement";  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var omn\_pagename="MainPage";  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var omn\_language="en";  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var omn\_newpagename="yes";  
&nbsp;&nbsp;&nbsp;&nbsp;\</script\>

... way down below...

\<td class="colPlus" onclick="fireClick(this,0)"\>  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;\<a id="iroc\_0" class="plus" href="#" onclick="return  
false;"\>&nbsp;\</a\>  
\</td\>

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Michael\_Fellinger1](https://avatars.discourse-cdn.com/v4/letter/m/e47774/32.png) [@Michael\_Fellinger1](https://rubytalk.org/u/Michael_Fellinger1)\
**Post date:** [20 May 2010 17:15 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/5 "2010-05-20T17:15:08Z")

</div>

> The other trick here is that this page is behind a login. Mechanize  
> allows me to fill out the login form and holds onto the login  
> credentials for me. Can harmony/celebrity/watir do this?

Watir definitely does that since it simply controls your browser and  
therefore behaves exactly like one.

> **···**
>
> On Fri, May 21, 2010 at 1:14 AM, Phil Mcdonnell \<phil.a.mcdonnell@gmail.com\> wrote:
> 
> > > The \*really\* interesting part is what does the Javascript do 🙂 with  
> > > (a potentially large) effort you may be able to "reverse-engineer" the  
> > > javascript and emulate manually in mechanize. I.e. if the javascript  
> > > builds a simple HTTP request, you may be able to send the same request  
> > > from mechanize (possibly) without much effort.
> > 
> > How would one do this? I'm somewhat new to javascript as I usually  
> > don't do front end engineering. I see the below definition of this  
> > function in the HTML page. Any way I can sniff out what it's actually  
> > doing? I'm looking to figure out what the fireClick method displays.
> > 
> > \<script type="text/javascript"\>  
> > var d = document.domain.split(".");  
> > document.domain = d[d.length - 2] + "." + d[d.length - 1];  
> > var start = (new Date()).getTime();  
> > var fireClick = function(){};  
> > var omn\_hierarchy="US|AMEX|Ser|eStatement";  
> > var omn\_pagename="MainPage";  
> > var omn\_language="en";  
> > var omn\_newpagename="yes";  
> > \</script\>
> > 
> > ... way down below...
> > 
> > \<td class="colPlus" onclick="fireClick(this,0)"\>  
> > \<a id="iroc\_0" class="plus" href="#" onclick="return  
> > false;"\>&nbsp;\</a\>  
> > \</td\>  
> > --  
> > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).
> 
> --  
> Michael Fellinger  
> CTO, The Rubyists, LLC  
> 972-996-5199

---

<div class="post-metadata">

**Author:** ![David\_Wright](https://avatars.discourse-cdn.com/v4/letter/d/c77e96/32.png) [@David\_Wright](https://rubytalk.org/u/David_Wright)\
**Post date:** [20 May 2010 17:19 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/6 "2010-05-20T17:19:02Z")

</div>

Mechanize cannot execute javascript but watir/celerity can. (I've never  
used harmony)

#in watir (could also use firewatir and/or the safari equivalent)  
require 'watir'  
require 'watir/ie'

# should work identically with celerity  
#require 'celerity  
#@browser = Celerity::IE.new

@login\_page = '[http://example.com/](http://example.com/)'

@browser = Watir::IE.new  
@browser.goto @login\_page  
@browser.text\_field(:name, 'username').set(@user)  
@browser.text\_field(:name, 'password').set(@pass)  
@browser.button(:value, "LogIn").click

# go to page where the javascript link is  
@broswer.link(:text, "Link Name").click

# click it  
# this assumes the fireClick event is 'just' an ajax call which returns  
content  
@broswer.link(:id, "iroc\_0").click  
@browser.wait # wait for ajax to return

# show page's displaying text (not view source)  
puts @browser.text  
# if above fires a pop up window more code is needed to retrieve the  
content

> **···**
>
> --  
> Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/).

---

<div class="post-metadata">

**Author:** ![Steven\_Parkes](https://avatars.discourse-cdn.com/v4/letter/s/6de8d8/32.png) [@Steven\_Parkes](https://rubytalk.org/u/Steven_Parkes)\
**Post date:** [20 May 2010 17:42 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/7 "2010-05-20T17:42:15Z")

</div>

> Mechanize cannot execute javascript but watir/celerity can. (I've never  
> used harmony)

Harmony uses envjs to execute JavaScript. There's also capybara which can either use a browser or envjs.

---

<div class="post-metadata">

**Author:** ![Phil\_Mcdonnell](https://avatars.discourse-cdn.com/v4/letter/p/e95f7d/32.png) [@Phil\_Mcdonnell](https://rubytalk.org/u/Phil_Mcdonnell)\
**Post date:** [24 May 2010 07:36 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/8 "2010-05-24T07:36:15Z")

</div>

This is extremely helpful!

With Watir I'm running into a problem finding the image button for login  
on the following page:  
[https://online.americanexpress.com/myca/logon/us/action?request\_type=LogonHandler&Face=en\_US&DestPage=https%3A%2F%2Fwww99.americanexpress.com%2Fmyca%2Facctsumm%2Fus%2Faction%3Frequest\_type%3Dauthreg\_acctAccountSummary%26us\_nu%3Dlogincontrol](https://online.americanexpress.com/myca/logon/us/action?request_type=LogonHandler&Face=en_US&DestPage=https%3A%2F%2Fwww99.americanexpress.com%2Fmyca%2Facctsumm%2Fus%2Faction%3Frequest_type%3Dauthreg_acctAccountSummary%26us_nu%3Dlogincontrol)

It looks like the login button is just a clickable image and I should be  
able to find it via:  
browser.button(:alt, "Login").click

Any idea why that doesn't find the button?

David Wright wrote:

> **···**
>
> > Mechanize cannot execute javascript but watir/celerity can. (I've never  
> > used harmony)
> > 
> > #in watir (could also use firewatir and/or the safari equivalent)  
> > require 'watir'  
> > require 'watir/ie'
> > 
> > # should work identically with celerity  
> > #require 'celerity  
> > #@browser = Celerity::IE.new
> > 
> > @login\_page = '[http://example.com/&#39](http://example.com/&#39);
> > 
> > @browser = Watir::IE.new  
> > @browser.goto @login\_page  
> > @browser.text\_field(:name, 'username').set(@user)  
> > @browser.text\_field(:name, 'password').set(@pass)  
> > @browser.button(:value, "LogIn").click
> > 
> > # go to page where the javascript link is  
> > @broswer.link(:text, "Link Name").click
> > 
> > # click it  
> > # this assumes the fireClick event is 'just' an ajax call which returns  
> > content  
> > @broswer.link(:id, "iroc\_0").click  
> > @browser.wait # wait for ajax to return
> > 
> > # show page's displaying text (not view source)  
> > puts @browser.text  
> > # if above fires a pop up window more code is needed to retrieve the  
> > content
> 
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).

---

<div class="post-metadata">

**Author:** ![Brabuhr](https://avatars.discourse-cdn.com/v4/letter/b/919ad9/32.png) [@Brabuhr](https://rubytalk.org/u/Brabuhr)\
**Post date:** [24 May 2010 14:32 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/9 "2010-05-24T14:32:36Z")

</div>

Sorry, don't have time to look at the page right now, but if it "is  
just a clickable image" and not an actual "button" watir's button  
helper may not find it (even though it looks like a button) so try  
browser.image().click?

> **···**
>
> On Mon, May 24, 2010 at 3:36 AM, Phil Mcdonnell \<phil.a.mcdonnell@gmail.com\> wrote:
> 
> > With Watir I'm running into a problem finding the image button for login  
> > on the following page:  
> > [One App](https://online.americanexpress.com/myca/logon/us/action?request_type=LogonHandler&Face=en_US&DestPage=https%3A%2F%2Fwww99.americanexpress.com%2Fmyca%2Facctsumm%2Fus%2Faction%3Frequest_type%3Dauthreg_acctAccountSummary%26us_nu%3Dlogincontrol)
> > 
> > It looks like the login button is just a clickable image and I should be  
> > able to find it via:  
> > browser.button(:alt, "Login").click
> > 
> > Any idea why that doesn't find the button?

---

<div class="post-metadata">

**Author:** ![Darryl\_gem\_dandy\_Bro](https://avatars.discourse-cdn.com/v4/letter/d/f6c823/32.png) [@Darryl\_gem\_dandy\_Bro](https://rubytalk.org/u/Darryl_gem_dandy_Bro)\
**Post date:** [26 May 2010 19:45 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/10 "2010-05-26T19:45:09Z")

</div>

To click on this with Watir:  
You can use:

&nbsp;&nbsp;@browser.button(:src, '[https://online.americanexpress.com/myca/logon/](https://online.americanexpress.com/myca/logon/)  
us/shared/images/btn\_login.gif').click

This was captured using the Webmetrics script recorder  
[http://www.webmetrics.com/products/script\_recorder.html](http://www.webmetrics.com/products/script_recorder.html)  
It has a Watir compatible mode. You won't get a working  
script out of it but it good for identifying objects.

Inspect Element using FireBug:

\<input type="image" border="0" onclick="javascript:loginNow();return  
false;" tabindex="5" style="margin-top: 5px; margin-left: 20px; margin-  
bottom: 22px;" alt="Login" src="/myca/logon/us/shared/images/  
btn\_login.gif"\>

A nice helper tool for identify page object such as this Webmetrics

Good luck,  
Darryl

> **···**
>
> On May 24, 10:32 am, brab...@gmail.com wrote:
> 
> > On Mon, May 24, 2010 at 3:36 AM, Phil Mcdonnell \> \> \<phil.a.mcdonn...@gmail.com\> wrote:  
> > \> With Watir I'm running into a problem finding the image button for login  
> > \> on the following page:  
> > \>[https://online.americanexpress.com/myca/logon/us/action?request\_type=\.\.\](https://online.americanexpress.com/myca/logon/us/action?request_type=%5C.%5C.%5C).
> > 
> > \> It looks like the login button is just a clickable image and I should be  
> > \> able to find it via:  
> > \> browser.button(:alt, "Login").click
> > 
> > \> Any idea why that doesn't find the button?
> > 
> > Sorry, don't have time to look at the page right now, but if it "is  
> > just a clickable image" and not an actual "button" watir's button  
> > helper may not find it (even though it looks like a button) so try  
> > browser.image().click?

---

<div class="post-metadata">

**Author:** ![Phil\_Mcdonnell](https://avatars.discourse-cdn.com/v4/letter/p/e95f7d/32.png) [@Phil\_Mcdonnell](https://rubytalk.org/u/Phil_Mcdonnell)\
**Post date:** [27 May 2010 05:29 UTC](https://rubytalk.org/t/scrape-javascript-content/58599/11 "2010-05-27T05:29:21Z")

</div>

Darryl! You just made my day! This does work. I've been banging my  
head on the wall for a while here 🙂 I had tried looking for the src  
tag too, but not with the full path (only the referential path in the  
html).

Thank you!

Darryl Brown wrote:

> **···**
>
> > To click on this with Watir:  
> > You can use:
> > 
> > &nbsp;&nbsp;@browser.button(:src, '[https://online.americanexpress.com/myca/logon/](https://online.americanexpress.com/myca/logon/)  
> > us/shared/images/btn\_login.gif').click
> > 
> > This was captured using the Webmetrics script recorder  
> > [http://www.webmetrics.com/products/script\_recorder.html](http://www.webmetrics.com/products/script_recorder.html)  
> > It has a Watir compatible mode. You won't get a working  
> > script out of it but it good for identifying objects.
> > 
> > Inspect Element using FireBug:
> > 
> > \<input type="image" border="0" onclick="javascript:loginNow();return  
> > false;" tabindex="5" style="margin-top: 5px; margin-left: 20px; margin-  
> > bottom: 22px;" alt="Login" src="/myca/logon/us/shared/images/  
> > btn\_login.gif"\>
> > 
> > A nice helper tool for identify page object such as this Webmetrics
> > 
> > Good luck,  
> > Darryl
> 
> --  
> Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C).