Posting over https

Andrew_Stewart · 9 October 2006 15:04

Hello Rubyists,

I am struggling to post data over https and would be grateful for any help.

I know the server works because I can hit it with curl and get the response I expect:

curl -d <url-encoded form data> url

When I try to do the same from my Ruby code the server doesn't like it.

The URL to hit looks like:

https://some.host.com/foo/bar.asp?service=someservice

My code looks like:

require 'net/https'
require 'uri'

   url = <as above>
   uri = URI.parse(url)
   request = Net::HTTP.new(uri.host, uri.port)
   request.use_ssl = true
   request.verify_mode = OpenSSL::SSL::VERIFY_NONE
   data = { 'key1' => 'value1',
            'key2' => 'value2',
            'keyn' => 'valuen' }.map {|k,v| "#{URI.escape(k.to_s)}=#{URI.escape(v.to_s)}"}.join('&')
   response = request.post("#{uri.path}?#{uri.query}", data)

When I tried this with curl, I used the value of data as my url-encoded form data and everything was fine.

Any ideas?

Thanks in advance,
Andy Stewart

Aaron_Patterson2 · 9 October 2006 15:52

Try using Mechanize. You can post forms over https with it, and you
won't have to deal with encoding query parameters or setting up
Net::HTTP.

http://mechanize.rubyforge.org/

--Aaron

···

On Tue, Oct 10, 2006 at 12:04:26AM +0900, Andrew Stewart wrote:

Hello Rubyists,

I am struggling to post data over https and would be grateful for any
help.

I know the server works because I can hit it with curl and get the
response I expect:

curl -d <url-encoded form data> url

When I try to do the same from my Ruby code the server doesn't like it.

The URL to hit looks like:

https://some.host.com/foo/bar.asp?service=someservice

My code looks like:

  require 'net/https'
  require 'uri'

  url = <as above>
  uri = URI.parse(url)
  request = Net::HTTP.new(uri.host, uri.port)
  request.use_ssl = true
  request.verify_mode = OpenSSL::SSL::VERIFY_NONE
  data = { 'key1' => 'value1',
           'key2' => 'value2',
           'keyn' => 'valuen' }.map {|k,v| "#{URI.escape
(k.to_s)}=#{URI.escape(v.to_s)}"}.join('&')
  response = request.post("#{uri.path}?#{uri.query}", data)

When I tried this with curl, I used the value of data as my url-
encoded form data and everything was fine.

Any ideas?

--
Aaron Patterson
http://tenderlovemaking.com/

Ara.T.Howard6 · 9 October 2006 17:52

fortytwo :/var/www/html > cat form.cgi
#! /usr/local/bin/ruby
require 'cgi'
require 'yaml'
(c=CGI.new).out("type" => "text/plain"){ c.params.to_yaml }

fortytwo :/var/www/html > cat post.rb
require 'cgi'
require 'net/https'
require 'uri'

url = ARGV.shift || "https://fortytwo.merseine.nu/form.cgi"
uri = URI.parse url

data = {
   'key1' => 'value1',
   'key2' => 'value2',
   'keyn' => 'valuen',
}

e = lambda{|x| CGI.escape x}
q = lambda{|h| h.map{|k,v| [e[k], e[v]].join '='}.join('&')}
form = q[data]

class HTTPS < Net::HTTP
   def initialize *a, &b
     super
     self.use_ssl = true
     self.verify_mode = OpenSSL::SSL::VERIFY_NONE
   end
end

HTTPS.start(uri.host, uri.port) do |https|
res = https.post uri.path, form
puts res.body
end

fortytwo :/var/www/html > ruby post.rb https://fortytwo.merseine.nu/form.cgi

···

On Tue, 10 Oct 2006, Andrew Stewart wrote:

Hello Rubyists,

I am struggling to post data over https and would be grateful for any help.

I know the server works because I can hit it with curl and get the response I expect:

curl -d <url-encoded form data> url

When I try to do the same from my Ruby code the server doesn't like it.

The URL to hit looks like:

https://some.host.com/foo/bar.asp?service=someservice

My code looks like:

require 'net/https'
require 'uri'

url = <as above>
uri = URI.parse(url)
request = Net::HTTP.new(uri.host, uri.port)
request.use_ssl = true
request.verify_mode = OpenSSL::SSL::VERIFY_NONE
data = { 'key1' => 'value1',
'key2' => 'value2',
'keyn' => 'valuen' }.map {|k,v| "#{URI.escape (k.to_s)}=#{URI.escape(v.to_s)}"}.join('&')
response = request.post("#{uri.path}?#{uri.query}", data)

When I tried this with curl, I used the value of data as my url-encoded form data and everything was fine.

Any ideas?

Thanks in advance,
Andy Stewart

---
key1:
- value1
key2:
- value2
keyn:
- valuen

-a
--
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei

Andrew_Stewart · 9 October 2006 16:53

Hi,

I am struggling to post data over https and would be grateful for any
help.

<snip/>

Try using Mechanize. You can post forms over https with it, and you
won't have to deal with encoding query parameters or setting up
Net::HTTP.

OK, but I would still like to understand what's wrong with my code so I can learn and improve.

Thanks anyway for the pointer,

Andy Stewart

Andrew_Stewart · 10 October 2006 13:16

Hello again,

fortytwo :/var/www/html > ruby post.rb https://fortytwo.merseine.nu/form.cgi
---
key1:
- value1
key2:
- value2
keyn:
- valuen

Many thanks for the pointers to Mechanize, Hpricot and this fully-fledged code sample. I do appreciate your taking the time to respond, and so quickly.

Adapting my code in the light of Ara's code above, with 'self.set_debug_output $stderr' added to the HTTPS initialize method, I compared the client-server conversation with curl's, which I knew worked. This led me to discover that the server expected the content-type header to be set to 'application/x-www-form-urlencoded'.

For the record, here's the code which works with the server I am talking to (Protx's payment gateway). Now that I have it working, I'll take Aaron and Richard's advice and look into replacing it with higher-level code using Mechanize.

require 'net/https'
require 'uri'

class HTTPS < Net::HTTP
   def initialize *a, &b
     super
     self.use_ssl = true
     self.verify_mode = OpenSSL::SSL::VERIFY_NONE
     #self.set_debug_output $stderr
   end
end

url = 'https://some.host.com/foo/bar.asp?service=someservice'
uri = URI.parse url
data = {
   'key1' => 'value1',
   'key2' => 'value2',
   'keyn' => 'valuen'
}
e = lambda {|x| URI.escape x}
q = lambda {|h| h.map {|k,v| [e[k], e[v]].join '='}.join('&')}
form = q[data]
HTTPS.start(uri.host, uri.port) do |https|
   response = https.post("#{uri.path}?#{uri.query}", form, {'content-type' => 'application/x-www-form-urlencoded'})
   puts response.body
end

Regards,
Andy Stewart

Richard_Conroy1 · 9 October 2006 17:01

Its admirable, but I think Net::HTTP is a bit too low level, and requires a
bit too much HTTP knowledge for many applications. Mechanize seems
to be the business, and possibly Hpricot might also have similar
functionality.

I found Net::HTTP a bit too frustrating to use and the lack of good examples
bothered me a lot. It seems that the real work along these lines (programmatic
HTTP/HTML processing) is done with Mechanize & Hpricot.

···

On 10/9/06, Andrew Stewart <boss@airbladesoftware.com> wrote:

Hi,

>> I am struggling to post data over https and would be grateful for any
>> help.

<snip/>

> Try using Mechanize. You can post forms over https with it, and you
> won't have to deal with encoding query parameters or setting up
> Net::HTTP.

OK, but I would still like to understand what's wrong with my code so
I can learn and improve.

Forum · 10 October 2006 14:26

code, now I'll just adapt your's (thx Ara)
Robert

···

On 10/10/06, Andrew Stewart <boss@airbladesoftware.com> wrote:

Hello again,

> fortytwo :/var/www/html > ruby post.rb https://fortytwo.merseine.nu/
> form.cgi
> ---
> key1:
> - value1
> key2:
> - value2
> keyn:
> - valuen

Many thanks for the pointers to Mechanize, Hpricot and this fully-
fledged code sample. I do appreciate your taking the time to
respond, and so quickly.

Adapting my code in the light of Ara's code above, with
'self.set_debug_output $stderr' added to the HTTPS initialize method,
I compared the client-server conversation with curl's, which I knew
worked. This led me to discover that the server expected the content-
type header to be set to 'application/x-www-form-urlencoded'.

For the record, here's the code which works with the server I am
talking to (Protx's payment gateway). Now that I have it working,
I'll take Aaron and Richard's advice and look into replacing it with
higher-level code using Mechanize.

require 'net/https'
require 'uri'

class HTTPS < Net::HTTP
   def initialize *a, &b
     super
     self.use_ssl = true
     self.verify_mode = OpenSSL::SSL::VERIFY_NONE
     #self.set_debug_output $stderr
   end
end

url = 'https://some.host.com/foo/bar.asp?service=someservice'
uri = URI.parse url
data = {
   'key1' => 'value1',
   'key2' => 'value2',
   'keyn' => 'valuen'
}
e = lambda {|x| URI.escape x}
q = lambda {|h| h.map {|k,v| [e[k], e[v]].join '='}.join('&')}
form = q[data]
HTTPS.start(uri.host, uri.port) do |https|
   response = https.post("#{uri.path}?#{uri.query}", form, {'content-
type' => 'application/x-www-form-urlencoded'})
   puts response.body
end

Regards,
Andy Stewart

Thanks for sharing this, I could not help you out as I lost my old POST

--
Deux choses sont infinies : l'univers et la bêtise humaine ; en ce qui
concerne l'univers, je n'en ai pas acquis la certitude absolue.

- Albert Einstein

Topic		Replies	Views
How can I post form data via https? ruby-talk	2	137	25 September 2008
Ssl encryption with post_form() ruby-talk	3	157	5 November 2010
HTTP request, with headers, post data and SSL :( ruby-talk	4	155	14 April 2011
Https post with xml data? ruby-talk	2	126	8 August 2009
Post xml over https connection ruby-talk	5	107	15 August 2007

Posting over https

Related topics