Using net/HTTP - How to login and then retrieve data?

ruby-talk
1

I'm new to Ruby, coming from Java. I need to write an HTTP client in
Ruby. I'm working in a Windows environment with Ruby 1.8 and RubyMine.

I've been working with net/http, and have written the code below, to
login to an HTTP server, then issue a POST to pull over some data, and
then logout. I can successfully Login, but the subsequent POST returns
an error indicating that I have Invalid Username/Password. I am also new
to HTTP processing, so I am not sure of the sequence of events that need
to take place for a successful Login-POST-Logout.

Please review my code and let me know what I need to do for the POST to
work.

Thanks!

--Bob

require 'net/http'

#LOGIN
url1 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/login')
req = Net::HTTP::Get.new(url1.path)
req.basic_auth 'myuser', 'mypass'
req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res1 = Net::HTTP.start(url1.host, url1.port) {|http| http.request(req)}
puts res1.body # This returns a 200 response and a listing of methods
that can be called, therefore this Login snippet works

#POST
url2Str = 'http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search'
url2 = URI.parse(url2Str)
path = 'http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search'
http = Net::HTTP.new(url2.host, 80)
data =
'SearchType=Property&Limit=15000&Format=COMPACT&Query=%28L_UpdateDate%3D2011-01-03T23:49:57%2B%29,%28L_UpdateDate%3D2011-01-04T23:49:57%2D%29&Class=BC_4'
headers = {
  'Accept' => '*/*',
  'User-Agent' => 'Mozilla/4.0',
  'RETS-Version' => 'RETS/1.7'
}
res2 = http.post(url2Str, data, headers)
puts res2.body # This returns error <Net::HTTPUnauthorized 401 Invalid
Username/Password combination readbody=true>

#LOGOUT
url3 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/logout')
req = Net::HTTP::Get.new(url3.path)
req.basic_auth 'myuser', 'mypass'
req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res3 = Net::HTTP.start(url3.host, url3.port) {|http| http.request(req)}
puts res3.body # This returns a 200 response

···

--
Posted via http://www.ruby-forum.com/.

2

I'm new to Ruby, coming from Java. I need to write an HTTP client in
Ruby. I'm working in a Windows environment with Ruby 1.8 and RubyMine.

I've been working with net/http, and have written the code below, to
login to an HTTP server, then issue a POST to pull over some data, and
then logout. I can successfully Login, but the subsequent POST returns
an error indicating that I have Invalid Username/Password. I am also new
to HTTP processing, so I am not sure of the sequence of events that need
to take place for a successful Login-POST-Logout.

Please review my code and let me know what I need to do for the POST to
work.

Thanks!

--Bob

require 'net/http'

#LOGIN
url1 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/login&#39;\)
req = Net::HTTP::Get.new(url1.path)
req.basic_auth 'myuser', 'mypass'

Do you need to also set .basic_auth on the POST request?
Do you need to store a cookie? (In which case, you might prefer using httpclient gem rather than net/http directly.)

You can also do:

     http = Net::HTTP.new(url.host, url.port)
     http.read_timeout = 120
     if url.scheme == 'https' || url.port == 443
       http.use_ssl = true
       http.verify_mode = OpenSSL::SSL::VERIFY_NONE
       http.timeout = 120
     end
     http.start do |connection|
       request = Net::HTTP::Post.new(url.request_uri)
       request.basic_auth 'user', 'password'

       request.body = 'something if needed'

       response = connection.request(request)

       # do other stuff with the same connection...
     end

req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res1 = Net::HTTP.start(url1.host, url1.port) {|http| http.request(req)}
puts res1.body # This returns a 200 response and a listing of methods
that can be called, therefore this Login snippet works

#POST
url2Str = 'http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search&#39;
url2 = URI.parse(url2Str)
path = 'http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search&#39;
http = Net::HTTP.new(url2.host, 80)
data =
'SearchType=Property&Limit=15000&Format=COMPACT&Query=%28L_UpdateDate%3D2011-01-03T23:49:57%2B%29,%28L_UpdateDate%3D2011-01-04T23:49:57%2D%29&Class=BC_4'
headers = {
'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0',
'RETS-Version' => 'RETS/1.7'
}
res2 = http.post(url2Str, data, headers)
puts res2.body # This returns error <Net::HTTPUnauthorized 401 Invalid
Username/Password combination readbody=true>

#LOGOUT
url3 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/logout&#39;\)
req = Net::HTTP::Get.new(url3.path)
req.basic_auth 'myuser', 'mypass'
req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res3 = Net::HTTP.start(url3.host, url3.port) {|http| http.request(req)}
puts res3.body # This returns a 200 response

--
Posted via http://www.ruby-forum.com/\.

Rob Biedenharn
Rob@AgileConsultingLLC.com http://AgileConsultingLLC.com/
rab@GaslightSoftware.com http://GaslightSoftware.com/

···

On Jan 21, 2011, at 2:20 PM, Bob C. wrote:

3

Well Net::HTTP is a bit on the low level side. You really need to know the
details of HTTP in order to be productive.

There are a lot of alternative HTTP client libraries out there that may give
you what you want a lot easier (abstracting the HTTP details away);
OTOH:
RestClient
HttpClient
Patron
Httparty
Resourceful
All of those are pure Ruby, and should work fine on windows.
For pure web scraping/form interacting clients you can take it up a level.
WATIR is well worth a look if you are on windows (your QA guys will want a
demo, as it has a visual mode)
There is also Webrat and Capybara which are outstanding, normally used for
assertion level testing.

If Java is your thing, you might want to consider JRuby specific options as
well, like Celerity (which uses native Java HttpClient IIRC)

Net::HTTP forces you down into a very low level of HTTP mechanics. For most
use cases this is not necessary, and its a tricky library to work with if
your needs are simpler.
Many of those other options are easier to learn and may be more directly
applicable to your problem (logging in, interacting & leaving)

···

On Fri, Jan 21, 2011 at 7:20 PM, Bob C. <rgc3679@yahoo.com> wrote:

I'm new to Ruby, coming from Java. I need to write an HTTP client in
Ruby. I'm working in a Windows environment with Ruby 1.8 and RubyMine.

I've been working with net/http, and have written the code below, to
login to an HTTP server, then issue a POST to pull over some data, and
then logout. I can successfully Login, but the subsequent POST returns
an error indicating that I have Invalid Username/Password. I am also new
to HTTP processing, so I am not sure of the sequence of events that need
to take place for a successful Login-POST-Logout.

--
http://richardconroy.blogspot.com | http://twitter.com/RichardConroy

4

Yes, POST also requires basic_auth.

No, a cookie is not required.

For now I'd like to stick with using net/http, and possibly later look
at things like httpclient.

When I code according to your example (see below) I am still failing on
the SEARCH operation due to error <Net::HTTPPreconditionFailed 412
Precondition Failed-User not logged in. readbody=true>. This doesn't
make sense because I have successfully logged in, and I am re-using the
connection.

I appreciate your help Rob, because I didn't know how to re-use the
connection - Thanks!

--Bob

Here's my current code:

require 'net/http'

url1 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/login&#39;\)
http = Net::HTTP.new(url1.host, url1.port)
http.read_timeout = 120
http.start do |connection|

  #LOGIN
  request1 = Net::HTTP::Post.new(url1.request_uri)
  request1.basic_auth 'myuser', 'mypass'
  request1.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
'*/*', 'User-Agent' => 'Mozilla/4.0'})
  #request1.body = 'something if needed'
  response1 = connection.request(request1)
  puts response1.body

  #SEARCH
  url2 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/search&#39;\)
  request2 = Net::HTTP::Post.new(url2.request_uri)
  request2.basic_auth 'myuser', 'mypass'
  request2.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
'*/*', 'User-Agent' => 'Mozilla/4.0',
  'SearchType' => 'Property',
  'Limit' => '15000',
  'Format' => 'COMPACT',
  'Query' =>
'%28L_UpdateDate%3D2011-01-03T23:49:57%2B%29,%28L_UpdateDate%3D2011-01-04T23:49:57%2D%29',
  'Class' => 'BC_4'
  })
  response2 = connection.request(request2)
  puts response2.inspect

  #LOGOUT
  url3 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/logout&#39;\)
  request3 = Net::HTTP::Post.new(url3.request_uri)
  request3.set_form_data({'RETS-Version' => 'RETS/1.7', 'Accept' =>
'*/*', 'User-Agent' => 'Mozilla/4.0'})
  request3.basic_auth 'myuser', 'mypass'
  response3 = connection.request(request3)
  puts response3.body
end
#<Net::HTTPOK 200 OK readbody=true>

#<Net::HTTPPreconditionFailed 412 Precondition Failed-User not logged
in. readbody=true>

#<Net::HTTPOK 200 OK readbody=true>

Rob Biedenharn wrote in post #976647:

···

On Jan 21, 2011, at 2:20 PM, Bob C. wrote:

to take place for a successful Login-POST-Logout.

#LOGIN
url1 =
URI.parse('http://imls.rets.fnismls.com/rets/fnisrets.aspx/imls/
login')
req = Net::HTTP::Get.new(url1.path)
req.basic_auth 'myuser', 'mypass'

Do you need to also set .basic_auth on the POST request?
Do you need to store a cookie? (In which case, you might prefer using
httpclient gem rather than net/http directly.)

You can also do:

     http = Net::HTTP.new(url.host, url.port)
     http.read_timeout = 120
     if url.scheme == 'https' || url.port == 443
       http.use_ssl = true
       http.verify_mode = OpenSSL::SSL::VERIFY_NONE
       http.timeout = 120
     end
     http.start do |connection|
       request = Net::HTTP::Post.new(url.request_uri)
       request.basic_auth 'user', 'password'

       request.body = 'something if needed'

       response = connection.request(request)

       # do other stuff with the same connection...
     end

url2 = URI.parse(url2Str)
}
req.set_form_data({"RETS-Version" => "RETS/1.7", 'Accept' => '*/*',
'User-Agent' => 'Mozilla/4.0'})
res3 = Net::HTTP.start(url3.host, url3.port) {|http|
http.request(req)}
puts res3.body # This returns a 200 response

--
Posted via http://www.ruby-forum.com/\.

Rob Biedenharn
Rob@AgileConsultingLLC.com http://AgileConsultingLLC.com/
rab@GaslightSoftware.com http://GaslightSoftware.com/

--
Posted via http://www.ruby-forum.com/\.

5

http://railstips.org/blog/archives/2008/07/29/it-s-an-httparty-and-everyone-is-invited/

6

soldier.coder wrote in post #976863:

http://railstips.org/blog/archives/2008/07/29/it-s-an-httparty-and-everyone-is-invited/

Thanks for that - looks like an interesting gem. I will try it out.

BTW: I have solved my problem - was using an encoded "Query" param that
was screwing up the server processing.

--Bob

···

--
Posted via http://www.ruby-forum.com/\.