Web Scraper - built with ruby and rails

ruby-talk
1

Hello Everyone,

As a budding rubyist and rails practitioner I am always looking for new
ways to apply the things I have been learning. Recently, I have become
increasingly interested in web scraping and I would like to build a web
application in rails that scrapes a particular data source. I was wondering
if anyone had any tutorials that combines both technologies (web scraping
and rails) I have spent the last few evenings after work combing the
internet for some but mostly they cover web scraping as a stand alone topic.

Any help would be great!

Thanks

Nathaniel

2

Hey Nathaniel!

Your app would probably be comprised of two parts: the scraper + the UI. I
know you want to use Ruby, but in the Python world there's Scrap [1] which
is a fairly mature scarping framework (I don't know what's available in
Ruby though). You can write the scraper in Python and the UI in Rails. I'm
not sure whether that'd work for you.

[1] http://scrapy.org/

Best regards

···

--
Greg Navis

On Tue, Jan 26, 2016 at 3:39 PM, Nathaniel Quashie <nathanielcfa@gmail.com> wrote:

Hello Everyone,

As a budding rubyist and rails practitioner I am always looking for new
ways to apply the things I have been learning. Recently, I have become
increasingly interested in web scraping and I would like to build a web
application in rails that scrapes a particular data source. I was wondering
if anyone had any tutorials that combines both technologies (web scraping
and rails) I have spent the last few evenings after work combing the
internet for some but mostly they cover web scraping as a stand alone topic.

Any help would be great!

Thanks

Nathaniel

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

3

I wrote a web scraper, it’s available here: GitHub - ioquatix/xapian-indexer: Xapian Indexer is an indexing spider written in Ruby and based on Xapian

Good luck.

···

On 27/01/2016, at 3:39 AM, Nathaniel Quashie <nathanielcfa@gmail.com> wrote:

Hello Everyone,

As a budding rubyist and rails practitioner I am always looking for new ways to apply the things I have been learning. Recently, I have become increasingly interested in web scraping and I would like to build a web application in rails that scrapes a particular data source. I was wondering if anyone had any tutorials that combines both technologies (web scraping and rails) I have spent the last few evenings after work combing the internet for some but mostly they cover web scraping as a stand alone topic.

Any help would be great!

Thanks
Nathaniel

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

4

Please share the link to the git repo when you want to share, people
would be happy to help with the actual app if it is public.
Darren Cauthon
darren@cauthon.com

···

On Tue, Jan 26, 2016 at 8:48 AM, Nathaniel Quashie <nathanielcfa@gmail.com> wrote:

Thanks guys for the quick replies!

Sam - i'll check out your application when I get out of the office.
Greg - I'd rather write everything in ruby, I prefer to write the code in
ruby.
Alvin - i'll check out that link later as well.

On Tue, Jan 26, 2016 at 9:43 AM, Alvin Dickson <dickson.alvin@gmail.com> > wrote:

You should check it out this gem.
GitHub - jaimeiniesta/metainspector: Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
They also have a demo app in rails.
https://metainspectordemo.herokuapp.com/

El mar., 26 de ene. de 2016 a la(s) 10:41, Samuel Williams >> <space.ship.traveller@gmail.com> escribió:

I wrote a web scraper, it’s available here:
GitHub - ioquatix/xapian-indexer: Xapian Indexer is an indexing spider written in Ruby and based on Xapian

Good luck.

On 27/01/2016, at 3:39 AM, Nathaniel Quashie <nathanielcfa@gmail.com> >>> wrote:

Hello Everyone,

As a budding rubyist and rails practitioner I am always looking for new
ways to apply the things I have been learning. Recently, I have become
increasingly interested in web scraping and I would like to build a web
application in rails that scrapes a particular data source. I was wondering
if anyone had any tutorials that combines both technologies (web scraping
and rails) I have spent the last few evenings after work combing the
internet for some but mostly they cover web scraping as a stand alone topic.

Any help would be great!

Thanks

Nathaniel

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--
--
Alvin Dickson
dickson.alvin@gmail.com

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

5

Thanks guys for the quick replies!

Sam - i'll check out your application when I get out of the office.
Greg - I'd rather write everything in ruby, I prefer to write the code in
ruby.
Alvin - i'll check out that link later as well.

···

On Tue, Jan 26, 2016 at 9:43 AM, Alvin Dickson <dickson.alvin@gmail.com> wrote:

You should check it out this gem.
GitHub - jaimeiniesta/metainspector: Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
They also have a demo app in rails.
https://metainspectordemo.herokuapp.com/

El mar., 26 de ene. de 2016 a la(s) 10:41, Samuel Williams < > space.ship.traveller@gmail.com> escribió:

I wrote a web scraper, it’s available here:
GitHub - ioquatix/xapian-indexer: Xapian Indexer is an indexing spider written in Ruby and based on Xapian

Good luck.

On 27/01/2016, at 3:39 AM, Nathaniel Quashie <nathanielcfa@gmail.com> >> wrote:

Hello Everyone,

As a budding rubyist and rails practitioner I am always looking for new
ways to apply the things I have been learning. Recently, I have become
increasingly interested in web scraping and I would like to build a web
application in rails that scrapes a particular data source. I was wondering
if anyone had any tutorials that combines both technologies (web scraping
and rails) I have spent the last few evenings after work combing the
internet for some but mostly they cover web scraping as a stand alone topic.

Any help would be great!

Thanks

Nathaniel

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe
<ruby-talk-request@ruby-lang.org?subject=unsubscribe>>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--
--
Alvin Dickson
dickson.alvin@gmail.com

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

6

You should check it out this gem.

They also have a demo app in rails. https://metainspectordemo.herokuapp.com/

···

El mar., 26 de ene. de 2016 a la(s) 10:41, Samuel Williams < space.ship.traveller@gmail.com> escribió:

I wrote a web scraper, it’s available here:
GitHub - ioquatix/xapian-indexer: Xapian Indexer is an indexing spider written in Ruby and based on Xapian

Good luck.

On 27/01/2016, at 3:39 AM, Nathaniel Quashie <nathanielcfa@gmail.com> > wrote:

Hello Everyone,

As a budding rubyist and rails practitioner I am always looking for new
ways to apply the things I have been learning. Recently, I have become
increasingly interested in web scraping and I would like to build a web
application in rails that scrapes a particular data source. I was wondering
if anyone had any tutorials that combines both technologies (web scraping
and rails) I have spent the last few evenings after work combing the
internet for some but mostly they cover web scraping as a stand alone topic.

Any help would be great!

Thanks

Nathaniel

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe
<ruby-talk-request@ruby-lang.org?subject=unsubscribe>>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--
--
Alvin Dickson
dickson.alvin@gmail.com

7

I created a web scraping application with Rails few months back which made
use of

1) GitHub - jnunemaker/httparty: 🎉 Makes http fun again! - for making http requests

2) GitHub - sparklemotion/nokogiri: Nokogiri (鋸) makes it easy and painless to work with XML and HTML from Ruby. - Parse HTML

3) GitHub - watir/watir-webdriver: watir-webdriver code has moved - web interactions like clicks
buttons etc

4) Postgresql as db

Thanks,
Prem

···

On Tue, Jan 26, 2016 at 8:54 PM, Darren Cauthon <darren@cauthon.com> wrote:

Please share the link to the git repo when you want to share, people
would be happy to help with the actual app if it is public.
Darren Cauthon
darren@cauthon.com

On Tue, Jan 26, 2016 at 8:48 AM, Nathaniel Quashie > <nathanielcfa@gmail.com> wrote:
> Thanks guys for the quick replies!
>
> Sam - i'll check out your application when I get out of the office.
> Greg - I'd rather write everything in ruby, I prefer to write the code in
> ruby.
> Alvin - i'll check out that link later as well.
>
> On Tue, Jan 26, 2016 at 9:43 AM, Alvin Dickson <dickson.alvin@gmail.com> > > wrote:
>>
>> You should check it out this gem.
>> GitHub - jaimeiniesta/metainspector: Ruby gem for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, links, images...
>> They also have a demo app in rails.
>> https://metainspectordemo.herokuapp.com/
>>
>> El mar., 26 de ene. de 2016 a la(s) 10:41, Samuel Williams > >> <space.ship.traveller@gmail.com> escribió:
>>>
>>> I wrote a web scraper, it’s available here:
>>> GitHub - ioquatix/xapian-indexer: Xapian Indexer is an indexing spider written in Ruby and based on Xapian
>>>
>>> Good luck.
>>>
>>>
>>>
>>>
>>> On 27/01/2016, at 3:39 AM, Nathaniel Quashie <nathanielcfa@gmail.com> > >>> wrote:
>>>
>>> Hello Everyone,
>>>
>>>
>>>
>>> As a budding rubyist and rails practitioner I am always looking for new
>>> ways to apply the things I have been learning. Recently, I have become
>>> increasingly interested in web scraping and I would like to build a web
>>> application in rails that scrapes a particular data source. I was
wondering
>>> if anyone had any tutorials that combines both technologies (web
scraping
>>> and rails) I have spent the last few evenings after work combing the
>>> internet for some but mostly they cover web scraping as a stand alone
topic.
>>>
>>>
>>>
>>> Any help would be great!
>>>
>>>
>>>
>>> Thanks
>>>
>>> Nathaniel
>>>
>>>
>>>
>>> Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
>>> <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;
>>>
>>>
>>>
>>> Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
>>> <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;
>>
>> --
>> --
>> Alvin Dickson
>> dickson.alvin@gmail.com
>>
>>
>> Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
>> <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;
>>
>
>
>
> Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
> <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;
>

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

8

Scrapy still doesn't work with Python 3 so not something I would recommend.

gvim

···

On 26/01/2016 14:43, Greg Navis wrote:

Hey Nathaniel!

Your app would probably be comprised of two parts: the scraper + the UI.
I know you want to use Ruby, but in the Python world there's Scrap [1]
which is a fairly mature scarping framework (I don't know what's
available in Ruby though). You can write the scraper in Python and the
UI in Rails. I'm not sure whether that'd work for you.

[1] http://scrapy.org/

Best regards
--
Greg Navis

On Tue, Jan 26, 2016 at 3:39 PM, Nathaniel Quashie > <nathanielcfa@gmail.com <mailto:nathanielcfa@gmail.com>> wrote:

    Hello Everyone,

    As a budding rubyist and rails practitioner I am always looking for
    new ways to apply the things I have been learning. Recently, I have
    become increasingly interested in web scraping and I would like to
    build a web application in rails that scrapes a particular data
    source. I was wondering if anyone had any tutorials that combines
    both technologies (web scraping and rails) I have spent the last few
    evenings after work combing the internet for some but mostly they
    cover web scraping as a stand alone topic.

    Any help would be great!

    Thanks

    Nathaniel

    Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
    <mailto:ruby-talk-request@ruby-lang.org>?subject=unsubscribe>
    <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

9

I have been following small tutorials since last night, I used this YouTube
tutorial that combines Nokogiri with Sinatra. I haven't completed the web
application part of it all, but I definitely getting a better understanding
of screen scraping (terrible name btw I thought it scraped screen
images....yeaaa....lol). I built a small simple scraper that scrapes the "
www.930.com/concerts" listings and displays the artist name, time of the
event, price of the ticket ( "sold out" if its sold out). I wish there were
more tuts out there around building web scraper web applications but
experimenting has been fun. As the project progresses I’ll be sure to post.

If anyone comes across any interesting blog posts or tuts around the
subject please alert me.

···

On Wed, Jan 27, 2016 at 9:23 AM, gvim <gvimrc@gmail.com> wrote:

Scrapy still doesn't work with Python 3 so not something I would recommend.

gvim

On 26/01/2016 14:43, Greg Navis wrote:

Hey Nathaniel!

Your app would probably be comprised of two parts: the scraper + the UI.
I know you want to use Ruby, but in the Python world there's Scrap [1]
which is a fairly mature scarping framework (I don't know what's
available in Ruby though). You can write the scraper in Python and the
UI in Rails. I'm not sure whether that'd work for you.

[1] http://scrapy.org/

Best regards
--
Greg Navis

On Tue, Jan 26, 2016 at 3:39 PM, Nathaniel Quashie >> <nathanielcfa@gmail.com <mailto:nathanielcfa@gmail.com>> wrote:

    Hello Everyone,

    As a budding rubyist and rails practitioner I am always looking for
    new ways to apply the things I have been learning. Recently, I have
    become increasingly interested in web scraping and I would like to
    build a web application in rails that scrapes a particular data
    source. I was wondering if anyone had any tutorials that combines
    both technologies (web scraping and rails) I have spent the last few
    evenings after work combing the internet for some but mostly they
    cover web scraping as a stand alone topic.

    Any help would be great!

    Thanks

    Nathaniel

    Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
    <mailto:ruby-talk-request@ruby-lang.org>?subject=unsubscribe>
    <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

10

You can use https://github.com/jeroenvandijk/capybara-mechanize with rails
and create your own scraper.

···

ср, 27 янв. 2016 г. в 18:14, Nathaniel Quashie <nathanielcfa@gmail.com>:

I have been following small tutorials since last night, I used this
YouTube tutorial that combines Nokogiri with Sinatra. I haven't completed
the web application part of it all, but I definitely getting a better
understanding of screen scraping (terrible name btw I thought it scraped
screen images....yeaaa....lol). I built a small simple scraper that scrapes
the "www.930.com/concerts" listings and displays the artist name, time of
the event, price of the ticket ( "sold out" if its sold out). I wish there
were more tuts out there around building web scraper web applications but
experimenting has been fun. As the project progresses I’ll be sure to post.

If anyone comes across any interesting blog posts or tuts around the
subject please alert me.

On Wed, Jan 27, 2016 at 9:23 AM, gvim <gvimrc@gmail.com> wrote:

Scrapy still doesn't work with Python 3 so not something I would
recommend.

gvim

On 26/01/2016 14:43, Greg Navis wrote:

Hey Nathaniel!

Your app would probably be comprised of two parts: the scraper + the UI.
I know you want to use Ruby, but in the Python world there's Scrap [1]
which is a fairly mature scarping framework (I don't know what's
available in Ruby though). You can write the scraper in Python and the
UI in Rails. I'm not sure whether that'd work for you.

[1] http://scrapy.org/

Best regards
--
Greg Navis

On Tue, Jan 26, 2016 at 3:39 PM, Nathaniel Quashie >>> <nathanielcfa@gmail.com <mailto:nathanielcfa@gmail.com>> wrote:

    Hello Everyone,

    As a budding rubyist and rails practitioner I am always looking for
    new ways to apply the things I have been learning. Recently, I have
    become increasingly interested in web scraping and I would like to
    build a web application in rails that scrapes a particular data
    source. I was wondering if anyone had any tutorials that combines
    both technologies (web scraping and rails) I have spent the last few
    evenings after work combing the internet for some but mostly they
    cover web scraping as a stand alone topic.

    Any help would be great!

    Thanks

    Nathaniel

    Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
    <mailto:ruby-talk-request@ruby-lang.org>?subject=unsubscribe>
    <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

11

Hey guys,

I completed a simple scraper for a concert venue website, the code works
locally but doesn't work once deployed to heroku. Does anyone know why the
code doesnt work on heroku vs locally?

···

On Thu, Jan 28, 2016 at 1:20 AM, Алексей Кудряшов < kudryashov.alexey.v@gmail.com> wrote:

You can use GitHub - phillbaker/capybara-mechanize: RackTest driver for Capybara with remote request support with
rails and create your own scraper.

ср, 27 янв. 2016 г. в 18:14, Nathaniel Quashie <nathanielcfa@gmail.com>:

I have been following small tutorials since last night, I used this
YouTube tutorial that combines Nokogiri with Sinatra. I haven't completed
the web application part of it all, but I definitely getting a better
understanding of screen scraping (terrible name btw I thought it scraped
screen images....yeaaa....lol). I built a small simple scraper that scrapes
the "www.930.com/concerts" listings and displays the artist name, time
of the event, price of the ticket ( "sold out" if its sold out). I wish
there were more tuts out there around building web scraper web applications
but experimenting has been fun. As the project progresses I’ll be sure to
post.

If anyone comes across any interesting blog posts or tuts around the
subject please alert me.

On Wed, Jan 27, 2016 at 9:23 AM, gvim <gvimrc@gmail.com> wrote:

Scrapy still doesn't work with Python 3 so not something I would
recommend.

gvim

On 26/01/2016 14:43, Greg Navis wrote:

Hey Nathaniel!

Your app would probably be comprised of two parts: the scraper + the UI.
I know you want to use Ruby, but in the Python world there's Scrap [1]
which is a fairly mature scarping framework (I don't know what's
available in Ruby though). You can write the scraper in Python and the
UI in Rails. I'm not sure whether that'd work for you.

[1] http://scrapy.org/

Best regards
--
Greg Navis

On Tue, Jan 26, 2016 at 3:39 PM, Nathaniel Quashie >>>> <nathanielcfa@gmail.com <mailto:nathanielcfa@gmail.com>> wrote:

    Hello Everyone,

    As a budding rubyist and rails practitioner I am always looking for
    new ways to apply the things I have been learning. Recently, I have
    become increasingly interested in web scraping and I would like to
    build a web application in rails that scrapes a particular data
    source. I was wondering if anyone had any tutorials that combines
    both technologies (web scraping and rails) I have spent the last few
    evenings after work combing the internet for some but mostly they
    cover web scraping as a stand alone topic.

    Any help would be great!

    Thanks

    Nathaniel

    Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
    <mailto:ruby-talk-request@ruby-lang.org>?subject=unsubscribe>
    <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

12

Hi Nathaniel,

Have you tried using Heroku's logging feature?

Ryan

···

On Sat, Jan 30, 2016 at 10:57 PM, Nathaniel Quashie <nathanielcfa@gmail.com> wrote:

Hey guys,

I completed a simple scraper for a concert venue website, the code works
locally but doesn't work once deployed to heroku. Does anyone know why the
code doesnt work on heroku vs locally?

GitHub - bklynate/samplescraper: wrote a scraper for a concert listing website in dc

On Thu, Jan 28, 2016 at 1:20 AM, Алексей Кудряшов < > kudryashov.alexey.v@gmail.com> wrote:

You can use GitHub - phillbaker/capybara-mechanize: RackTest driver for Capybara with remote request support with
rails and create your own scraper.

ср, 27 янв. 2016 г. в 18:14, Nathaniel Quashie <nathanielcfa@gmail.com>:

I have been following small tutorials since last night, I used this
YouTube tutorial that combines Nokogiri with Sinatra. I haven't completed
the web application part of it all, but I definitely getting a better
understanding of screen scraping (terrible name btw I thought it scraped
screen images....yeaaa....lol). I built a small simple scraper that scrapes
the "www.930.com/concerts" listings and displays the artist name, time
of the event, price of the ticket ( "sold out" if its sold out). I wish
there were more tuts out there around building web scraper web applications
but experimenting has been fun. As the project progresses I’ll be sure to
post.

If anyone comes across any interesting blog posts or tuts around the
subject please alert me.

On Wed, Jan 27, 2016 at 9:23 AM, gvim <gvimrc@gmail.com> wrote:

Scrapy still doesn't work with Python 3 so not something I would
recommend.

gvim

On 26/01/2016 14:43, Greg Navis wrote:

Hey Nathaniel!

Your app would probably be comprised of two parts: the scraper + the
UI.
I know you want to use Ruby, but in the Python world there's Scrap [1]
which is a fairly mature scarping framework (I don't know what's
available in Ruby though). You can write the scraper in Python and the
UI in Rails. I'm not sure whether that'd work for you.

[1] http://scrapy.org/

Best regards
--
Greg Navis

On Tue, Jan 26, 2016 at 3:39 PM, Nathaniel Quashie >>>>> <nathanielcfa@gmail.com <mailto:nathanielcfa@gmail.com>> wrote:

    Hello Everyone,

    As a budding rubyist and rails practitioner I am always looking for
    new ways to apply the things I have been learning. Recently, I have
    become increasingly interested in web scraping and I would like to
    build a web application in rails that scrapes a particular data
    source. I was wondering if anyone had any tutorials that combines
    both technologies (web scraping and rails) I have spent the last
few
    evenings after work combing the internet for some but mostly they
    cover web scraping as a stand alone topic.

    Any help would be great!

    Thanks

    Nathaniel

    Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
    <mailto:ruby-talk-request@ruby-lang.org>?subject=unsubscribe>
    <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--
Ryan Cook
Simple and to everyplace but the point.
720.319.7660

13

Hey ryan,

Checked out the logs and what i see is a timeout of some sort, not really
sure how or why the application is timing out.

···

On Jan 31, 2016 9:51 AM, "Ryan Cook" <cookrn@gmail.com> wrote:

Hi Nathaniel,

Have you tried using Heroku's logging feature?
Logging | Heroku Dev Center

Ryan

On Sat, Jan 30, 2016 at 10:57 PM, Nathaniel Quashie < > nathanielcfa@gmail.com> wrote:

Hey guys,

I completed a simple scraper for a concert venue website, the code works
locally but doesn't work once deployed to heroku. Does anyone know why the
code doesnt work on heroku vs locally?

GitHub - bklynate/samplescraper: wrote a scraper for a concert listing website in dc

On Thu, Jan 28, 2016 at 1:20 AM, Алексей Кудряшов < >> kudryashov.alexey.v@gmail.com> wrote:

You can use GitHub - phillbaker/capybara-mechanize: RackTest driver for Capybara with remote request support with
rails and create your own scraper.

ср, 27 янв. 2016 г. в 18:14, Nathaniel Quashie <nathanielcfa@gmail.com>:

I have been following small tutorials since last night, I used this
YouTube tutorial that combines Nokogiri with Sinatra. I haven't completed
the web application part of it all, but I definitely getting a better
understanding of screen scraping (terrible name btw I thought it scraped
screen images....yeaaa....lol). I built a small simple scraper that scrapes
the "www.930.com/concerts" listings and displays the artist name, time
of the event, price of the ticket ( "sold out" if its sold out). I wish
there were more tuts out there around building web scraper web applications
but experimenting has been fun. As the project progresses I’ll be sure to
post.

If anyone comes across any interesting blog posts or tuts around the
subject please alert me.

On Wed, Jan 27, 2016 at 9:23 AM, gvim <gvimrc@gmail.com> wrote:

Scrapy still doesn't work with Python 3 so not something I would
recommend.

gvim

On 26/01/2016 14:43, Greg Navis wrote:

Hey Nathaniel!

Your app would probably be comprised of two parts: the scraper + the
UI.
I know you want to use Ruby, but in the Python world there's Scrap [1]
which is a fairly mature scarping framework (I don't know what's
available in Ruby though). You can write the scraper in Python and the
UI in Rails. I'm not sure whether that'd work for you.

[1] http://scrapy.org/

Best regards
--
Greg Navis

On Tue, Jan 26, 2016 at 3:39 PM, Nathaniel Quashie >>>>>> <nathanielcfa@gmail.com <mailto:nathanielcfa@gmail.com>> wrote:

    Hello Everyone,

    As a budding rubyist and rails practitioner I am always looking
for
    new ways to apply the things I have been learning. Recently, I
have
    become increasingly interested in web scraping and I would like to
    build a web application in rails that scrapes a particular data
    source. I was wondering if anyone had any tutorials that combines
    both technologies (web scraping and rails) I have spent the last
few
    evenings after work combing the internet for some but mostly they
    cover web scraping as a stand alone topic.

    Any help would be great!

    Thanks

    Nathaniel

    Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
    <mailto:ruby-talk-request@ruby-lang.org>?subject=unsubscribe>
    <http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

--
Ryan Cook
Simple and to everyplace but the point.
720.319.7660

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

14

"of some sort" - awesome description

No one else is likely to be "sure" either unless you provide some
actual information about the problem.

···

On Sun, Jan 31, 2016 at 7:57 AM, Nathaniel Quashie <nathanielcfa@gmail.com> wrote:

Checked out the logs and what i see is a timeout of some sort, not really
sure how or why the application is timing out.

--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com

twitter: @hassan
Consulting Availability : Silicon Valley or remote

15

The sarcastic smug response Hassan isn't needed, I'm sure there was a
better way of telling to be more descriptive without coming off as a total
wanker, nevertheless this is the heroku log:

2016-01-31T18:02:21.689529+00:00 app[web.1]: 2016-01-31 18:02:21 -
Net::ReadTimeout - Net::ReadTimeout:

2016-01-31T18:02:21.689534+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue
in rbuf_fill'

2016-01-31T18:02:21.689536+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:134:in
`readuntil'

2016-01-31T18:02:21.689537+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:144:in
`readline'

2016-01-31T18:02:21.689538+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:39:in
`read_status_line'

2016-01-31T18:02:21.689538+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:28:in
`read_new'

2016-01-31T18:02:21.689539+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1412:in `block in
transport_request'

2016-01-31T18:02:21.689540+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `catch'

2016-01-31T18:02:21.689541+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in
`transport_request'

2016-01-31T18:02:21.689541+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1382:in `request'

2016-01-31T18:02:21.689542+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:319:in `block in
open_http'

2016-01-31T18:02:21.689543+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:852:in `start'

2016-01-31T18:02:21.689543+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:313:in `open_http'

2016-01-31T18:02:21.689544+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:709:in
`buffer_open'

2016-01-31T18:02:21.689545+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:210:in `block in
open_loop'

2016-01-31T18:02:21.689545+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `catch'

2016-01-31T18:02:21.689546+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `open_loop'

2016-01-31T18:02:21.689546+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'

2016-01-31T18:02:21.689547+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:689:in `open'

2016-01-31T18:02:21.689547+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:34:in `open'

2016-01-31T18:02:21.689548+00:00 app[web.1]: simple_scrape.rb:7:in
`block in <main>'

2016-01-31T18:02:21.689549+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in
`call'

2016-01-31T18:02:21.689550+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in
`block in compile!'

2016-01-31T18:02:21.689551+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in
`'

2016-01-31T18:02:21.689551+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in
`block (3 levels) in route!'

2016-01-31T18:02:21.689571+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in
`block in route!'

2016-01-31T18:02:21.689572+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in
`each'

2016-01-31T18:02:21.689573+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in
`route!'

2016-01-31T18:02:21.689573+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in
`block in dispatch!'

2016-01-31T18:02:21.689574+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
`block in invoke'

2016-01-31T18:02:21.689575+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
`catch'

2016-01-31T18:02:21.689575+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
`invoke'

2016-01-31T18:02:21.689576+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in
`dispatch!'

2016-01-31T18:02:21.689576+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in
`block in call!'

2016-01-31T18:02:21.689577+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
`block in invoke'

2016-01-31T18:02:21.689577+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
`catch'

2016-01-31T18:02:21.689578+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
`invoke'

2016-01-31T18:02:21.689579+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in
`call!'

2016-01-31T18:02:21.689580+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in
`call'

2016-01-31T18:02:21.689580+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in
`call'

2016-01-31T18:02:21.689581+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in
`call'

2016-01-31T18:02:21.689582+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in
`call'

2016-01-31T18:02:21.689594+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in
`call'

2016-01-31T18:02:21.689595+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in
`call'

2016-01-31T18:02:21.689595+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in
`call'

2016-01-31T18:02:21.689596+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in
`call'

2016-01-31T18:02:21.689596+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in
`call'

2016-01-31T18:02:21.689597+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in
`call'

2016-01-31T18:02:21.689597+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in
`call'

2016-01-31T18:02:21.689598+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in
`call'

2016-01-31T18:02:21.689599+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in
`call'

2016-01-31T18:02:21.689600+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in
`block in call'

2016-01-31T18:02:21.689602+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/thin-1.6.4/lib/thin/connection.rb:84:in
`catch'

2016-01-31T18:02:21.689603+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/thin-1.6.4/lib/thin/connection.rb:84:in
`pre_process'

2016-01-31T18:02:21.689603+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in
`call'

2016-01-31T18:02:21.689604+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in
`block in spawn_threadpool'

2016-01-31T18:02:21.689818+00:00 app[web.1]: 100.38.74.239 - -
[31/Jan/2016:18:02:21 +0000] "GET / HTTP/1.1" 500 30 120.3564

2016-01-31T18:02:21.697138+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue
in rbuf_fill'

2016-01-31T18:02:21.697140+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:152:in
`rbuf_fill'

2016-01-31T18:02:21.697143+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:28:in
`read_new'

2016-01-31T18:02:21.697144+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1412:in `block in
transport_request'

2016-01-31T18:02:21.697145+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `catch'

2016-01-31T18:02:21.697145+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in
`transport_request'

2016-01-31T18:02:21.697148+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:852:in `start'

2016-01-31T18:02:21.697148+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:313:in `open_http'

2016-01-31T18:02:21.697155+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:210:in `block in
open_loop'

2016-01-31T18:02:21.697156+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `catch'

2016-01-31T18:02:21.697157+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'

2016-01-31T18:02:21.697157+00:00 app[web.1]:
  /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:689:in `open'

2016-01-31T18:02:21.697160+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in
`call'

2016-01-31T18:02:21.697161+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in
`block in compile!'

2016-01-31T18:02:21.697161+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in
`'

2016-01-31T18:02:21.697162+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in
`block (3 levels) in route!'

2016-01-31T18:02:21.697163+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in
`route_eval'

2016-01-31T18:02:21.697163+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in
`block (2 levels) in route!'

2016-01-31T18:02:21.697164+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in
`block in process_route'

2016-01-31T18:02:21.697164+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in
`catch'

2016-01-31T18:02:21.697165+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in
`process_route'

2016-01-31T18:02:21.697166+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in
`block in route!'

2016-01-31T18:02:21.697168+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in
`block in dispatch!'

2016-01-31T18:02:21.697168+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
`block in invoke'

2016-01-31T18:02:21.697173+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in
`dispatch!'

2016-01-31T18:02:21.697174+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in
`block in call!'

2016-01-31T18:02:21.697176+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in
`invoke'

2016-01-31T18:02:21.697176+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in
`call!'

2016-01-31T18:02:21.697300+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in
`call'

2016-01-31T18:02:21.697302+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in
`call'

2016-01-31T18:02:21.697303+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in
`call'

2016-01-31T18:02:21.697322+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in
`call'

2016-01-31T18:02:21.697323+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in
`block in call'

2016-01-31T18:02:21.697324+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in
`synchronize'

2016-01-31T18:02:21.697324+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in
`call'

2016-01-31T18:02:21.697327+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in
`call'

2016-01-31T18:02:21.697332+00:00 app[web.1]:
  /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.r

···

On Jan 31, 2016 12:10 PM, "Hassan Schroeder" <hassan.schroeder@gmail.com> wrote:

On Sun, Jan 31, 2016 at 7:57 AM, Nathaniel Quashie > <nathanielcfa@gmail.com> wrote:

> Checked out the logs and what i see is a timeout of some sort, not really
> sure how or why the application is timing out.

"of some sort" - awesome description

No one else is likely to be "sure" either unless you provide some
actual information about the problem.

--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com
Hassan Schroeder | about.me
twitter: @hassan
Consulting Availability : Silicon Valley or remote

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

16

Hey Nathaniel!

On the surface, it looks as if your code couldn't get data from the
specified URL.

I suggest that you run `heroku run irb` and experiment with the code on
Heroku. Also, a better architecture for your application might be a
separate scraper that stores the scraped concerts to the database + a
separate web UI that displays the scraped concerts.

Best regards

···

--
Greg Navis

On Sun, Jan 31, 2016 at 7:03 PM, Nathaniel Quashie <nathanielcfa@gmail.com> wrote:

The sarcastic smug response Hassan isn't needed, I'm sure there was a
better way of telling to be more descriptive without coming off as a total
wanker, nevertheless this is the heroku log:

2016-01-31T18:02:21.689529+00:00 app[web.1]: 2016-01-31 18:02:21 - Net::ReadTimeout - Net::ReadTimeout:

2016-01-31T18:02:21.689534+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue in rbuf_fill'

2016-01-31T18:02:21.689536+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'

2016-01-31T18:02:21.689537+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'

2016-01-31T18:02:21.689538+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'

2016-01-31T18:02:21.689538+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'

2016-01-31T18:02:21.689539+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1412:in `block in transport_request'

2016-01-31T18:02:21.689540+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `catch'

2016-01-31T18:02:21.689541+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `transport_request'

2016-01-31T18:02:21.689541+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1382:in `request'

2016-01-31T18:02:21.689542+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:319:in `block in open_http'

2016-01-31T18:02:21.689543+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:852:in `start'

2016-01-31T18:02:21.689543+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:313:in `open_http'

2016-01-31T18:02:21.689544+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:709:in `buffer_open'

2016-01-31T18:02:21.689545+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop'

2016-01-31T18:02:21.689545+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `catch'

2016-01-31T18:02:21.689546+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `open_loop'

2016-01-31T18:02:21.689546+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'

2016-01-31T18:02:21.689547+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:689:in `open'

2016-01-31T18:02:21.689547+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:34:in `open'

2016-01-31T18:02:21.689548+00:00 app[web.1]: simple_scrape.rb:7:in `block in <main>'

2016-01-31T18:02:21.689549+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'

2016-01-31T18:02:21.689550+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'

2016-01-31T18:02:21.689551+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `'

2016-01-31T18:02:21.689551+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'

2016-01-31T18:02:21.689571+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'

2016-01-31T18:02:21.689572+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'

2016-01-31T18:02:21.689573+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'

2016-01-31T18:02:21.689573+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'

2016-01-31T18:02:21.689574+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'

2016-01-31T18:02:21.689575+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'

2016-01-31T18:02:21.689575+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'

2016-01-31T18:02:21.689576+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'

2016-01-31T18:02:21.689576+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'

2016-01-31T18:02:21.689577+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'

2016-01-31T18:02:21.689577+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'

2016-01-31T18:02:21.689578+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'

2016-01-31T18:02:21.689579+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'

2016-01-31T18:02:21.689580+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'

2016-01-31T18:02:21.689580+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'

2016-01-31T18:02:21.689581+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'

2016-01-31T18:02:21.689582+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'

2016-01-31T18:02:21.689594+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'

2016-01-31T18:02:21.689595+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'

2016-01-31T18:02:21.689595+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'

2016-01-31T18:02:21.689596+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'

2016-01-31T18:02:21.689596+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'

2016-01-31T18:02:21.689597+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in `call'

2016-01-31T18:02:21.689597+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'

2016-01-31T18:02:21.689598+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'

2016-01-31T18:02:21.689599+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'

2016-01-31T18:02:21.689600+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'

2016-01-31T18:02:21.689602+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/thin-1.6.4/lib/thin/connection.rb:84:in `catch'

2016-01-31T18:02:21.689603+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/thin-1.6.4/lib/thin/connection.rb:84:in `pre_process'

2016-01-31T18:02:21.689603+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `call'

2016-01-31T18:02:21.689604+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `block in spawn_threadpool'

2016-01-31T18:02:21.689818+00:00 app[web.1]: 100.38.74.239 - - [31/Jan/2016:18:02:21 +0000] "GET / HTTP/1.1" 500 30 120.3564

2016-01-31T18:02:21.697138+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue in rbuf_fill'

2016-01-31T18:02:21.697140+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill'

2016-01-31T18:02:21.697143+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'

2016-01-31T18:02:21.697144+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1412:in `block in transport_request'

2016-01-31T18:02:21.697145+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `catch'

2016-01-31T18:02:21.697145+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `transport_request'

2016-01-31T18:02:21.697148+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:852:in `start'

2016-01-31T18:02:21.697148+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:313:in `open_http'

2016-01-31T18:02:21.697155+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop'

2016-01-31T18:02:21.697156+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `catch'

2016-01-31T18:02:21.697157+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'

2016-01-31T18:02:21.697157+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:689:in `open'

2016-01-31T18:02:21.697160+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'

2016-01-31T18:02:21.697161+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'

2016-01-31T18:02:21.697161+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `'

2016-01-31T18:02:21.697162+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'

2016-01-31T18:02:21.697163+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'

2016-01-31T18:02:21.697163+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'

2016-01-31T18:02:21.697164+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'

2016-01-31T18:02:21.697164+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'

2016-01-31T18:02:21.697165+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'

2016-01-31T18:02:21.697166+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'

2016-01-31T18:02:21.697168+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'

2016-01-31T18:02:21.697168+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'

2016-01-31T18:02:21.697173+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'

2016-01-31T18:02:21.697174+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'

2016-01-31T18:02:21.697176+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'

2016-01-31T18:02:21.697176+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'

2016-01-31T18:02:21.697300+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'

2016-01-31T18:02:21.697302+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'

2016-01-31T18:02:21.697303+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'

2016-01-31T18:02:21.697322+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'

2016-01-31T18:02:21.697323+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'

2016-01-31T18:02:21.697324+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'

2016-01-31T18:02:21.697324+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'

2016-01-31T18:02:21.697327+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `call'

2016-01-31T18:02:21.697332+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.r

On Jan 31, 2016 12:10 PM, "Hassan Schroeder" <hassan.schroeder@gmail.com> > wrote:

On Sun, Jan 31, 2016 at 7:57 AM, Nathaniel Quashie >> <nathanielcfa@gmail.com> wrote:

> Checked out the logs and what i see is a timeout of some sort, not
really
> sure how or why the application is timing out.

"of some sort" - awesome description

No one else is likely to be "sure" either unless you provide some
actual information about the problem.

--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com
Hassan Schroeder | about.me
twitter: @hassan
Consulting Availability : Silicon Valley or remote

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

17

Hi Greg,

Thanks for your feedback, I found a little bit of time to tinker this code
last night and still haven't found a answer. I use a pry binding in my code
and it revealed the all the variables are present and have the right
information stored in them. For the life of me, I cant figure out why this
code isn't working on heroku, yet runs fine locally. I'm assuming the issue
may line within some heroku deployment I may have overlooked.

my code is here <https://github.com/bklynate/samplescraper&gt; - it may be
alot to ask but could someone pull the code and try to deploy it themselves
and see if they encounter the same issue?

-Nathaniel

···

On Tue, Feb 2, 2016 at 4:07 AM, Greg Navis <contact@gregnavis.com> wrote:

Hey Nathaniel!

On the surface, it looks as if your code couldn't get data from the
specified URL.

I suggest that you run `heroku run irb` and experiment with the code on
Heroku. Also, a better architecture for your application might be a
separate scraper that stores the scraped concerts to the database + a
separate web UI that displays the scraped concerts.

Best regards
--
Greg Navis

On Sun, Jan 31, 2016 at 7:03 PM, Nathaniel Quashie <nathanielcfa@gmail.com > > wrote:

The sarcastic smug response Hassan isn't needed, I'm sure there was a
better way of telling to be more descriptive without coming off as a total
wanker, nevertheless this is the heroku log:

2016-01-31T18:02:21.689529+00:00 app[web.1]: 2016-01-31 18:02:21 - Net::ReadTimeout - Net::ReadTimeout:

2016-01-31T18:02:21.689534+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue in rbuf_fill'

2016-01-31T18:02:21.689536+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:134:in `readuntil'

2016-01-31T18:02:21.689537+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:144:in `readline'

2016-01-31T18:02:21.689538+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:39:in `read_status_line'

2016-01-31T18:02:21.689538+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'

2016-01-31T18:02:21.689539+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1412:in `block in transport_request'

2016-01-31T18:02:21.689540+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `catch'

2016-01-31T18:02:21.689541+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `transport_request'

2016-01-31T18:02:21.689541+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1382:in `request'

2016-01-31T18:02:21.689542+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:319:in `block in open_http'

2016-01-31T18:02:21.689543+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:852:in `start'

2016-01-31T18:02:21.689543+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:313:in `open_http'

2016-01-31T18:02:21.689544+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:709:in `buffer_open'

2016-01-31T18:02:21.689545+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop'

2016-01-31T18:02:21.689545+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `catch'

2016-01-31T18:02:21.689546+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `open_loop'

2016-01-31T18:02:21.689546+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'

2016-01-31T18:02:21.689547+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:689:in `open'

2016-01-31T18:02:21.689547+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:34:in `open'

2016-01-31T18:02:21.689548+00:00 app[web.1]: simple_scrape.rb:7:in `block in <main>'

2016-01-31T18:02:21.689549+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'

2016-01-31T18:02:21.689550+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'

2016-01-31T18:02:21.689551+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `'

2016-01-31T18:02:21.689551+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'

2016-01-31T18:02:21.689571+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'

2016-01-31T18:02:21.689572+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `each'

2016-01-31T18:02:21.689573+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:972:in `route!'

2016-01-31T18:02:21.689573+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'

2016-01-31T18:02:21.689574+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'

2016-01-31T18:02:21.689575+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'

2016-01-31T18:02:21.689575+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'

2016-01-31T18:02:21.689576+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'

2016-01-31T18:02:21.689576+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'

2016-01-31T18:02:21.689577+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'

2016-01-31T18:02:21.689577+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `catch'

2016-01-31T18:02:21.689578+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'

2016-01-31T18:02:21.689579+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'

2016-01-31T18:02:21.689580+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in `call'

2016-01-31T18:02:21.689580+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in `call'

2016-01-31T18:02:21.689581+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in `call'

2016-01-31T18:02:21.689582+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'

2016-01-31T18:02:21.689594+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'

2016-01-31T18:02:21.689595+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'

2016-01-31T18:02:21.689595+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'

2016-01-31T18:02:21.689596+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/commonlogger.rb:33:in `call'

2016-01-31T18:02:21.689596+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:219:in `call'

2016-01-31T18:02:21.689597+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:212:in `call'

2016-01-31T18:02:21.689597+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/head.rb:13:in `call'

2016-01-31T18:02:21.689598+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/methodoverride.rb:22:in `call'

2016-01-31T18:02:21.689599+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'

2016-01-31T18:02:21.689600+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'

2016-01-31T18:02:21.689602+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/thin-1.6.4/lib/thin/connection.rb:84:in `catch'

2016-01-31T18:02:21.689603+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/thin-1.6.4/lib/thin/connection.rb:84:in `pre_process'

2016-01-31T18:02:21.689603+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `call'

2016-01-31T18:02:21.689604+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `block in spawn_threadpool'

2016-01-31T18:02:21.689818+00:00 app[web.1]: 100.38.74.239 - - [31/Jan/2016:18:02:21 +0000] "GET / HTTP/1.1" 500 30 120.3564

2016-01-31T18:02:21.697138+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:158:in `rescue in rbuf_fill'

2016-01-31T18:02:21.697140+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/protocol.rb:152:in `rbuf_fill'

2016-01-31T18:02:21.697143+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http/response.rb:28:in `read_new'

2016-01-31T18:02:21.697144+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1412:in `block in transport_request'

2016-01-31T18:02:21.697145+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `catch'

2016-01-31T18:02:21.697145+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:1409:in `transport_request'

2016-01-31T18:02:21.697148+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/net/http.rb:852:in `start'

2016-01-31T18:02:21.697148+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:313:in `open_http'

2016-01-31T18:02:21.697155+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop'

2016-01-31T18:02:21.697156+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:208:in `catch'

2016-01-31T18:02:21.697157+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'

2016-01-31T18:02:21.697157+00:00 app[web.1]: /app/vendor/ruby-2.0.0/lib/ruby/2.0.0/open-uri.rb:689:in `open'

2016-01-31T18:02:21.697160+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `call'

2016-01-31T18:02:21.697161+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1611:in `block in compile!'

2016-01-31T18:02:21.697161+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `'

2016-01-31T18:02:21.697162+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (3 levels) in route!'

2016-01-31T18:02:21.697163+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:994:in `route_eval'

2016-01-31T18:02:21.697163+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:975:in `block (2 levels) in route!'

2016-01-31T18:02:21.697164+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1015:in `block in process_route'

2016-01-31T18:02:21.697164+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `catch'

2016-01-31T18:02:21.697165+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1013:in `process_route'

2016-01-31T18:02:21.697166+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:973:in `block in route!'

2016-01-31T18:02:21.697168+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1085:in `block in dispatch!'

2016-01-31T18:02:21.697168+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `block in invoke'

2016-01-31T18:02:21.697173+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1082:in `dispatch!'

2016-01-31T18:02:21.697174+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `block in call!'

2016-01-31T18:02:21.697176+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1067:in `invoke'

2016-01-31T18:02:21.697176+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:907:in `call!'

2016-01-31T18:02:21.697300+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in `call'

2016-01-31T18:02:21.697302+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in `call'

2016-01-31T18:02:21.697303+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/rack-1.6.4/lib/rack/logger.rb:15:in `call'

2016-01-31T18:02:21.697322+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:2013:in `call'

2016-01-31T18:02:21.697323+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `block in call'

2016-01-31T18:02:21.697324+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1787:in `synchronize'

2016-01-31T18:02:21.697324+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/sinatra-1.4.7/lib/sinatra/base.rb:1487:in `call'

2016-01-31T18:02:21.697327+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:1067:in `call'

2016-01-31T18:02:21.697332+00:00 app[web.1]: /app/vendor/bundle/ruby/2.0.0/gems/eventmachine-1.0.9.1/lib/eventmachine.r

On Jan 31, 2016 12:10 PM, "Hassan Schroeder" <hassan.schroeder@gmail.com> >> wrote:

On Sun, Jan 31, 2016 at 7:57 AM, Nathaniel Quashie >>> <nathanielcfa@gmail.com> wrote:

> Checked out the logs and what i see is a timeout of some sort, not
really
> sure how or why the application is timing out.

"of some sort" - awesome description

No one else is likely to be "sure" either unless you provide some
actual information about the problem.

--
Hassan Schroeder ------------------------ hassan.schroeder@gmail.com
Hassan Schroeder | about.me
twitter: @hassan
Consulting Availability : Silicon Valley or remote

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org
?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;

Unsubscribe: <mailto:ruby-talk-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk&gt;