# Posting over https

**URL:** <https://rubytalk.org/t/posting-over-https/31721>\
**Category:** ruby-talk\
**Created:** [9 October 2006 15:04 UTC](https://rubytalk.org/t/posting-over-https/31721 "2006-10-09T15:04:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Andrew\_Stewart](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@Andrew\_Stewart](https://rubytalk.org/u/Andrew_Stewart)\
**Post date:** [9 October 2006 15:04 UTC](https://rubytalk.org/t/posting-over-https/31721/1 "2006-10-09T15:04:26Z")

</div>

Hello Rubyists,

I am struggling to post data over https and would be grateful for any help.

I know the server works because I can hit it with curl and get the response I expect:

curl -d \<url-encoded form data\> url

When I try to do the same from my Ruby code the server doesn't like it.

The URL to hit looks like:

[https://some.host.com/foo/bar.asp?service=someservice](https://some.host.com/foo/bar.asp?service=someservice)

My code looks like:

&nbsp;&nbsp;&nbsp;require 'net/https'  
&nbsp;&nbsp;&nbsp;require 'uri'

&nbsp;&nbsp;&nbsp;url = \<as above\>  
&nbsp;&nbsp;&nbsp;uri = URI.parse(url)  
&nbsp;&nbsp;&nbsp;request = Net::HTTP.new(uri.host, uri.port)  
&nbsp;&nbsp;&nbsp;request.use\_ssl = true  
&nbsp;&nbsp;&nbsp;request.verify\_mode = OpenSSL::SSL::VERIFY\_NONE  
&nbsp;&nbsp;&nbsp;data = { 'key1' =\> 'value1',  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'key2' =\> 'value2',  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'keyn' =\> 'valuen' }.map {|k,v| "#{URI.escape(k.to\_s)}=#{URI.escape(v.to\_s)}"}.join('&')  
&nbsp;&nbsp;&nbsp;response = request.post("#{uri.path}?#{uri.query}", data)

When I tried this with curl, I used the value of data as my url-encoded form data and everything was fine.

Any ideas?

Thanks in advance,  
Andy Stewart

---

<div class="post-metadata">

**Author:** ![Aaron\_Patterson2](https://avatars.discourse-cdn.com/v4/letter/a/e480ec/32.png) [@Aaron\_Patterson2](https://rubytalk.org/u/Aaron_Patterson2)\
**Post date:** [9 October 2006 15:52 UTC](https://rubytalk.org/t/posting-over-https/31721/2 "2006-10-09T15:52:36Z")

</div>

Try using Mechanize. You can post forms over https with it, and you  
won't have to deal with encoding query parameters or setting up  
Net::HTTP.

&nbsp;&nbsp;[http://mechanize.rubyforge.org/](http://mechanize.rubyforge.org/)

--Aaron

> **···**
>
> On Tue, Oct 10, 2006 at 12:04:26AM +0900, Andrew Stewart wrote:
> 
> > Hello Rubyists,
> > 
> > I am struggling to post data over https and would be grateful for any  
> > help.
> > 
> > I know the server works because I can hit it with curl and get the  
> > response I expect:
> > 
> > curl -d \<url-encoded form data\> url
> > 
> > When I try to do the same from my Ruby code the server doesn't like it.
> > 
> > The URL to hit looks like:
> > 
> > [https://some.host.com/foo/bar.asp?service=someservice](https://some.host.com/foo/bar.asp?service=someservice)
> > 
> > My code looks like:
> > 
> > &nbsp;&nbsp;require 'net/https'  
> > &nbsp;&nbsp;require 'uri'
> > 
> > &nbsp;&nbsp;url = \<as above\>  
> > &nbsp;&nbsp;uri = URI.parse(url)  
> > &nbsp;&nbsp;request = Net::HTTP.new(uri.host, uri.port)  
> > &nbsp;&nbsp;request.use\_ssl = true  
> > &nbsp;&nbsp;request.verify\_mode = OpenSSL::SSL::VERIFY\_NONE  
> > &nbsp;&nbsp;data = { 'key1' =\> 'value1',  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'key2' =\> 'value2',  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'keyn' =\> 'valuen' }.map {|k,v| "#{URI.escape  
> > (k.to\_s)}=#{URI.escape(v.to\_s)}"}.join('&')  
> > &nbsp;&nbsp;response = request.post("#{uri.path}?#{uri.query}", data)
> > 
> > When I tried this with curl, I used the value of data as my url-  
> > encoded form data and everything was fine.
> > 
> > Any ideas?
> 
> --  
> Aaron Patterson  
> [http://tenderlovemaking.com/](http://tenderlovemaking.com/)

---

<div class="post-metadata">

**Author:** ![Ara.T.Howard6](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Ara.T.Howard6](https://rubytalk.org/u/Ara.T.Howard6)\
**Post date:** [9 October 2006 17:52 UTC](https://rubytalk.org/t/posting-over-https/31721/3 "2006-10-09T17:52:11Z")

</div>

fortytwo :/var/www/html \> cat form.cgi  
#! /usr/local/bin/ruby  
require 'cgi'  
require 'yaml'  
(c=CGI.new).out("type" =\> "text/plain"){ c.params.to\_yaml }

fortytwo :/var/www/html \> cat post.rb  
require 'cgi'  
require 'net/https'  
require 'uri'

url = ARGV.shift || "[https://fortytwo.merseine.nu/form.cgi&quot](https://fortytwo.merseine.nu/form.cgi&quot);  
uri = URI.parse url

data = {  
&nbsp;&nbsp;&nbsp;'key1' =\> 'value1',  
&nbsp;&nbsp;&nbsp;'key2' =\> 'value2',  
&nbsp;&nbsp;&nbsp;'keyn' =\> 'valuen',  
}

e = lambda{|x| CGI.escape x}  
q = lambda{|h| h.map{|k,v| [e[k], e[v]].join '='}.join('&')}  
form = q[data]

class HTTPS \< Net::HTTP  
&nbsp;&nbsp;&nbsp;def initialize \*a, &b  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;super  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.use\_ssl = true  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.verify\_mode = OpenSSL::SSL::VERIFY\_NONE  
&nbsp;&nbsp;&nbsp;end  
end

HTTPS.start(uri.host, uri.port) do |https|  
&nbsp;&nbsp;&nbsp;res = https.post uri.path, form  
&nbsp;&nbsp;&nbsp;puts res.body  
end

fortytwo :/var/www/html \> ruby post.rb [https://fortytwo.merseine.nu/form.cgi](https://fortytwo.merseine.nu/form.cgi)

> **···**
>
> On Tue, 10 Oct 2006, Andrew Stewart wrote:
> 
> > Hello Rubyists,
> > 
> > I am struggling to post data over https and would be grateful for any help.
> > 
> > I know the server works because I can hit it with curl and get the response I expect:
> > 
> > curl -d \<url-encoded form data\> url
> > 
> > When I try to do the same from my Ruby code the server doesn't like it.
> > 
> > The URL to hit looks like:
> > 
> > [https://some.host.com/foo/bar.asp?service=someservice](https://some.host.com/foo/bar.asp?service=someservice)
> > 
> > My code looks like:
> > 
> > require 'net/https'  
> > require 'uri'
> > 
> > url = \<as above\>  
> > uri = URI.parse(url)  
> > request = Net::HTTP.new(uri.host, uri.port)  
> > request.use\_ssl = true  
> > request.verify\_mode = OpenSSL::SSL::VERIFY\_NONE  
> > data = { 'key1' =\> 'value1',  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'key2' =\> 'value2',  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;'keyn' =\> 'valuen' }.map {|k,v| "#{URI.escape (k.to\_s)}=#{URI.escape(v.to\_s)}"}.join('&')  
> > response = request.post("#{uri.path}?#{uri.query}", data)
> > 
> > When I tried this with curl, I used the value of data as my url-encoded form data and everything was fine.
> > 
> > Any ideas?
> > 
> > Thanks in advance,  
> > Andy Stewart
> 
> ---  
> key1:  
> - value1  
> key2:  
> - value2  
> keyn:  
> - valuen
> 
> -a  
> --  
> in order to be effective truth must penetrate like an arrow - and that is  
> likely to hurt. -- wei wu wei

---

<div class="post-metadata">

**Author:** ![Andrew\_Stewart](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@Andrew\_Stewart](https://rubytalk.org/u/Andrew_Stewart)\
**Post date:** [9 October 2006 16:53 UTC](https://rubytalk.org/t/posting-over-https/31721/4 "2006-10-09T16:53:38Z")

</div>

Hi,

> > I am struggling to post data over https and would be grateful for any  
> > help.

\<snip/\>

> Try using Mechanize. You can post forms over https with it, and you  
> won't have to deal with encoding query parameters or setting up  
> Net::HTTP.

OK, but I would still like to understand what's wrong with my code so I can learn and improve.

Thanks anyway for the pointer,

Andy Stewart

---

<div class="post-metadata">

**Author:** ![Andrew\_Stewart](https://avatars.discourse-cdn.com/v4/letter/a/9de053/32.png) [@Andrew\_Stewart](https://rubytalk.org/u/Andrew_Stewart)\
**Post date:** [10 October 2006 13:16 UTC](https://rubytalk.org/t/posting-over-https/31721/5 "2006-10-10T13:16:39Z")

</div>

Hello again,

> fortytwo :/var/www/html \> ruby post.rb [https://fortytwo.merseine.nu/form.cgi](https://fortytwo.merseine.nu/form.cgi)  
> ---  
> key1:  
> - value1  
> key2:  
> - value2  
> keyn:  
> - valuen

Many thanks for the pointers to Mechanize, Hpricot and this fully-fledged code sample. I do appreciate your taking the time to respond, and so quickly.

Adapting my code in the light of Ara's code above, with 'self.set\_debug\_output $stderr' added to the HTTPS initialize method, I compared the client-server conversation with curl's, which I knew worked. This led me to discover that the server expected the content-type header to be set to 'application/x-www-form-urlencoded'.

For the record, here's the code which works with the server I am talking to (Protx's payment gateway). Now that I have it working, I'll take Aaron and Richard's advice and look into replacing it with higher-level code using Mechanize.

require 'net/https'  
require 'uri'

class HTTPS \< Net::HTTP  
&nbsp;&nbsp;&nbsp;def initialize \*a, &b  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;super  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.use\_ssl = true  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.verify\_mode = OpenSSL::SSL::VERIFY\_NONE  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#self.set\_debug\_output $stderr  
&nbsp;&nbsp;&nbsp;end  
end

url = '[https://some.host.com/foo/bar.asp?service=someservice&#39](https://some.host.com/foo/bar.asp?service=someservice&#39);  
uri = URI.parse url  
data = {  
&nbsp;&nbsp;&nbsp;'key1' =\> 'value1',  
&nbsp;&nbsp;&nbsp;'key2' =\> 'value2',  
&nbsp;&nbsp;&nbsp;'keyn' =\> 'valuen'  
}  
e = lambda {|x| URI.escape x}  
q = lambda {|h| h.map {|k,v| [e[k], e[v]].join '='}.join('&')}  
form = q[data]  
HTTPS.start(uri.host, uri.port) do |https|  
&nbsp;&nbsp;&nbsp;response = https.post("#{uri.path}?#{uri.query}", form, {'content-type' =\> 'application/x-www-form-urlencoded'})  
&nbsp;&nbsp;&nbsp;puts response.body  
end

Regards,  
Andy Stewart

---

<div class="post-metadata">

**Author:** ![Richard\_Conroy1](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@Richard\_Conroy1](https://rubytalk.org/u/Richard_Conroy1)\
**Post date:** [9 October 2006 17:01 UTC](https://rubytalk.org/t/posting-over-https/31721/6 "2006-10-09T17:01:32Z")

</div>

Its admirable, but I think Net::HTTP is a bit too low level, and requires a  
bit too much HTTP knowledge for many applications. Mechanize seems  
to be the business, and possibly Hpricot might also have similar  
functionality.

I found Net::HTTP a bit too frustrating to use and the lack of good examples  
bothered me a lot. It seems that the real work along these lines (programmatic  
HTTP/HTML processing) is done with Mechanize & Hpricot.

> **···**
>
> On 10/9/06, Andrew Stewart \<boss@airbladesoftware.com\> wrote:
> 
> > Hi,
> > 
> > \>\> I am struggling to post data over https and would be grateful for any  
> > \>\> help.
> > 
> > \<snip/\>
> > 
> > \> Try using Mechanize. You can post forms over https with it, and you  
> > \> won't have to deal with encoding query parameters or setting up  
> > \> Net::HTTP.
> > 
> > OK, but I would still like to understand what's wrong with my code so  
> > I can learn and improve.

---

<div class="post-metadata">

**Author:** ![Forum](https://avatars.discourse-cdn.com/v4/letter/f/43a26b/32.png) [@Forum](https://rubytalk.org/u/Forum)\
**Post date:** [10 October 2006 14:26 UTC](https://rubytalk.org/t/posting-over-https/31721/7 "2006-10-10T14:26:03Z")

</div>

code, now I'll just adapt your's (thx Ara) 🙂  
Robert

> **···**
>
> On 10/10/06, Andrew Stewart \<boss@airbladesoftware.com\> wrote:
> 
> > Hello again,
> > 
> > \> fortytwo :/var/www/html \> ruby post.rb [https://fortytwo.merseine.nu/](https://fortytwo.merseine.nu/)  
> > \> form.cgi  
> > \> ---  
> > \> key1:  
> > \> - value1  
> > \> key2:  
> > \> - value2  
> > \> keyn:  
> > \> - valuen
> > 
> > Many thanks for the pointers to Mechanize, Hpricot and this fully-  
> > fledged code sample. I do appreciate your taking the time to  
> > respond, and so quickly.
> > 
> > Adapting my code in the light of Ara's code above, with  
> > 'self.set\_debug\_output $stderr' added to the HTTPS initialize method,  
> > I compared the client-server conversation with curl's, which I knew  
> > worked. This led me to discover that the server expected the content-  
> > type header to be set to 'application/x-www-form-urlencoded'.
> > 
> > For the record, here's the code which works with the server I am  
> > talking to (Protx's payment gateway). Now that I have it working,  
> > I'll take Aaron and Richard's advice and look into replacing it with  
> > higher-level code using Mechanize.
> > 
> > require 'net/https'  
> > require 'uri'
> > 
> > class HTTPS \< Net::HTTP  
> > &nbsp;&nbsp;&nbsp;def initialize \*a, &b  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;super  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.use\_ssl = true  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.verify\_mode = OpenSSL::SSL::VERIFY\_NONE  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#self.set\_debug\_output $stderr  
> > &nbsp;&nbsp;&nbsp;end  
> > end
> > 
> > url = '[https://some.host.com/foo/bar.asp?service=someservice&#39](https://some.host.com/foo/bar.asp?service=someservice&#39);  
> > uri = URI.parse url  
> > data = {  
> > &nbsp;&nbsp;&nbsp;'key1' =\> 'value1',  
> > &nbsp;&nbsp;&nbsp;'key2' =\> 'value2',  
> > &nbsp;&nbsp;&nbsp;'keyn' =\> 'valuen'  
> > }  
> > e = lambda {|x| URI.escape x}  
> > q = lambda {|h| h.map {|k,v| [e[k], e[v]].join '='}.join('&')}  
> > form = q[data]  
> > HTTPS.start(uri.host, uri.port) do |https|  
> > &nbsp;&nbsp;&nbsp;response = https.post("#{uri.path}?#{uri.query}", form, {'content-  
> > type' =\> 'application/x-www-form-urlencoded'})  
> > &nbsp;&nbsp;&nbsp;puts response.body  
> > end
> > 
> > Regards,  
> > Andy Stewart
> > 
> > Thanks for sharing this, I could not help you out as I lost my old POST
> 
> --  
> Deux choses sont infinies : l'univers et la bêtise humaine ; en ce qui  
> concerne l'univers, je n'en ai pas acquis la certitude absolue.
> 
> - Albert Einstein