Having trouble connecting to a site via net/http

ruby-talk
1

Hello,

   In my app I have the need to screen scrape a HTTPS page and I'm
getting a strange error that I don't understand. I can reproduce the
error with a secure google page, which I do in the test code below.
Any help would be much appreciated.

   Janak

#My Ruby version
ruby 1.8.4 (2005-12-24) [i386-mswin32]

# My test code
require 'net/http.rb'
require 'uri'

response =
Net::HTTP.get_response(URI.parse('https://www.google.com/adsense/'))

c:/ruby/lib/ruby/1.8/net/protocol.rb:133:in `sysread': Invalid argument
(Errno::EINVAL)
        from c:/ruby/lib/ruby/1.8/net/protocol.rb:133:in `rbuf_fill'
        from c:/ruby/lib/ruby/1.8/timeout.rb:56:in `timeout'
        from c:/ruby/lib/ruby/1.8/timeout.rb:76:in `timeout'
        from c:/ruby/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
        from c:/ruby/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
        from c:/ruby/lib/ruby/1.8/net/protocol.rb:126:in `readline'
        from c:/ruby/lib/ruby/1.8/net/http.rb:1988:in
`read_status_line'
        from c:/ruby/lib/ruby/1.8/net/http.rb:1977:in `read_new'
        from c:/ruby/lib/ruby/1.8/net/http.rb:1046:in `request'
        from c:/ruby/lib/ruby/1.8/net/http.rb:944:in `request_get'
        from c:/ruby/lib/ruby/1.8/net/http.rb:380:in `get_response'
        from c:/ruby/lib/ruby/1.8/net/http.rb:545:in `start'
        from c:/ruby/lib/ruby/1.8/net/http.rb:379:in `get_response'
        from adtest.rb:27

2

Use net/https to connect over ssl...

require 'net/https'
require 'uri'

uri = URI.parse('https://adwords.google.com')

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
# http.verify_mode = OpenSSL::SSL::VERIFY_NONE # Not setting this
explicitly will result in an error and the value being set anyway
response = http.get('/')
p response

···

On 10/20/06, jakattack2@gmail.com <jakattack2@gmail.com> wrote:

Hello,

   In my app I have the need to screen scrape a HTTPS page and I'm
getting a strange error that I don't understand. I can reproduce the
error with a secure google page, which I do in the test code below.
Any help would be much appreciated.

   Janak

#My Ruby version
ruby 1.8.4 (2005-12-24) [i386-mswin32]

# My test code
require 'net/http.rb'
require 'uri'

response =
Net::HTTP.get_response(URI.parse('https://www.google.com/adsense/&#39;\))

3

I tried the code that you suggest and I get the following error
messages. The error seems to be thrown on this line

response = http.get('/')

    Janak

# Error messages
c:/ruby/lib/ruby/1.8/net/http.rb:565:in `connect': undefined method
`verify_mode' for nil:NilClass (NoMethodError)
        from c:/ruby/lib/ruby/1.8/net/http.rb:555:in `do_start'
        from c:/ruby/lib/ruby/1.8/net/http.rb:544:in `start'
        from c:/ruby/lib/ruby/1.8/net/http.rb:1031:in `request'
        from c:/ruby/lib/ruby/1.8/net/http.rb:771:in `get'
        from adtest.rb:32

Chris Roos wrote:

···

On 10/20/06, jakattack2@gmail.com <jakattack2@gmail.com> wrote:
> Hello,
>
> In my app I have the need to screen scrape a HTTPS page and I'm
> getting a strange error that I don't understand. I can reproduce the
> error with a secure google page, which I do in the test code below.
> Any help would be much appreciated.
>
> Janak
>
> #My Ruby version
> ruby 1.8.4 (2005-12-24) [i386-mswin32]
>
> # My test code
> require 'net/http.rb'
> require 'uri'
>
> response =
> Net::HTTP.get_response(URI.parse('https://www.google.com/adsense/&#39;\))
>
Use net/https to connect over ssl...

require 'net/https'
require 'uri'

uri = URI.parse('https://adwords.google.com')

http = Net::HTTP.new(uri.host, uri.port)
http.use_ssl = true
# http.verify_mode = OpenSSL::SSL::VERIFY_NONE # Not setting this
explicitly will result in an error and the value being set anyway
response = http.get('/')
p response

4

Did you require net/https (NOT net/http)?

Chris

···

On 10/20/06, jakattack2@gmail.com <jakattack2@gmail.com> wrote:

I tried the code that you suggest and I get the following error
messages. The error seems to be thrown on this line

response = http.get('/')

    Janak

# Error messages
c:/ruby/lib/ruby/1.8/net/http.rb:565:in `connect': undefined method
`verify_mode' for nil:NilClass (NoMethodError)
        from c:/ruby/lib/ruby/1.8/net/http.rb:555:in `do_start'
        from c:/ruby/lib/ruby/1.8/net/http.rb:544:in `start'
        from c:/ruby/lib/ruby/1.8/net/http.rb:1031:in `request'
        from c:/ruby/lib/ruby/1.8/net/http.rb:771:in `get'
        from adtest.rb:32