# Http/net **URL:** \ **Category:** ruby-talk\ **Created:** [25 September 2002 16:03 UTC](https://rubytalk.org/t/http-net/1982 "2002-09-25T16:03:07Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Manfred\_Hansen1](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@Manfred\_Hansen1](https://rubytalk.org/u/Manfred_Hansen1)\ **Post date:** [25 September 2002 16:03 UTC](https://rubytalk.org/t/http-net/1982/1 "2002-09-25T16:03:07Z")
Hi, is this possible with http/net . data =`lynx http://linux-lx:80/ -auth=hansen:123456 -dump` puts data The page need a password and a user for login. Manfred ---
**Author:** ![ts1](https://avatars.discourse-cdn.com/v4/letter/t/a8b319/32.png) [@ts1](https://rubytalk.org/u/ts1)\ **Post date:** [25 September 2002 16:05 UTC](https://rubytalk.org/t/http-net/1982/2 "2002-09-25T16:05:53Z")
> The page need a password and a user for login. Well, if you use basic authentification you have in the documentation === Basic Authentication     require 'net/http'     Net::HTTP.version\_1\_1 # declear to use 1.1 features.     Net::HTTP.start( 'auth.some.domain' ) {|http|         response, body = http.get( '/need-auth.cgi',                 'Authorization' =\> 'Basic ' + ["#{account}:#{password}"].pack('m'). strip )         print body     } In version 1.2 (Ruby 1.7 or later), you can write like this:     require 'net/http'     Net::HTTP.version\_1\_2 # declear to use 1.2 features.     req = Net::HTTP::Get.new('/need-auth.cgi')     req.basic\_auth 'account', 'password'     Net::HTTP.start( 'auth.some.domain' ) {|http|         response = http.request(req)         print response.body     } Guy Decoux ---
**Author:** ![Manfred\_Hansen1](https://avatars.discourse-cdn.com/v4/letter/m/d78d45/32.png) [@Manfred\_Hansen1](https://rubytalk.org/u/Manfred_Hansen1)\ **Post date:** [25 September 2002 17:43 UTC](https://rubytalk.org/t/http-net/1982/3 "2002-09-25T17:43:17Z")
Thanks for your advice. Manfred Hansen ts wrote: > **···** > > > > The page need a password and a user for login. > > > > Well, if you use basic authentification you have in the documentation > > > > === Basic Authentication > > > > ``` > > require 'net/http' > > Net::HTTP.version_1_1 # declear to use 1.1 features. > > > > Net::HTTP.start( 'auth.some.domain' ) {|http| > > response, body = http.get( '/need-auth.cgi', > > 'Authorization' => 'Basic ' + > > ["#{account}:#{password}"].pack('m'). > > > > ``` > > > > strip ) > > print body > > } > > > > In version 1.2 (Ruby 1.7 or later), you can write like this: > > > > ``` > > require 'net/http' > > Net::HTTP.version_1_2 # declear to use 1.2 features. > > > > req = Net::HTTP::Get.new('/need-auth.cgi') > > req.basic_auth 'account', 'password' > > Net::HTTP.start( 'auth.some.domain' ) {|http| > > response = http.request(req) > > print response.body > > } > > > > ``` > > > > Guy Decoux ---
**Author:** ![Jim\_Freeze2](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@Jim\_Freeze2](https://rubytalk.org/u/Jim_Freeze2)\ **Post date:** [25 September 2002 18:20 UTC](https://rubytalk.org/t/http-net/1982/4 "2002-09-25T18:20:19Z")
First, let me say that I am a net/http dummy. I would like to collect information from a website at work. From my Netscape perspective, I type in the URL: [http://xyz.mycompany.com](http://xyz.mycompany.com) It automatically redirects to [http://xyz.mycompany.com:3000/scripts/isynch.dll](http://xyz.mycompany.com:3000/scripts/isynch.dll) If I have not logged on, I get a login popup dialogue asking for my username and password. I tried the following code, but it gave an error. Can someone point me in the right direction? > **···** > > On Thu, Sep 26, 2002 at 01:05:53AM +0900, ts wrote: > > > > The page need a password and a user for login. > > > > Well, if you use basic authentification you have in the documentation > > > > === Basic Authentication > > > > ``` > > require 'net/http' > > Net::HTTP.version_1_1 # declear to use 1.1 features. > > > > Net::HTTP.start( 'auth.some.domain' ) {|http| > > response, body = http.get( '/need-auth.cgi', > > 'Authorization' => 'Basic ' + ["#{account}:#{password}"].pack('m'). > > > > ``` > > > > strip ) > > print body > > } > > ================ > require ‘net/http’ > Net::HTTP.version\_1\_1 # declear to use 1.1 features. > > url0 = “[http://xyz.mycompany.com](http://xyz.mycompany.com)” > url = “[http://xyz.mycompany.com:3000/](http://xyz.mycompany.com:3000/)” > auth = “/scripts/isynch.dll” > account = “me” > passwd = “notmyrealpassword” > > # Net::HTTP.start( url ) {|http| response, body = http.get( auth, ‘Authorization’ =\> 'Basic ’ + [“#{account}:#{passwd}”].pack(‘m’).strip ) print body } > > Thanks > > Jim > > > In version 1.2 (Ruby 1.7 or later), you can write like this: > > > > ``` > > require 'net/http' > > Net::HTTP.version_1_2 # declear to use 1.2 features. > > > > req = Net::HTTP::Get.new('/need-auth.cgi') > > req.basic_auth 'account', 'password' > > Net::HTTP.start( 'auth.some.domain' ) {|http| > > response = http.request(req) > > print response.body > > } > > > > ``` > > > > Guy Decoux > > ## – Jim Freeze > > Programming Ruby > def initialize; fun; end > A language with class ---
**Author:** ![Jim\_Freeze2](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@Jim\_Freeze2](https://rubytalk.org/u/Jim_Freeze2)\ **Post date:** [25 September 2002 19:10 UTC](https://rubytalk.org/t/http-net/1982/5 "2002-09-25T19:10:34Z")
> I tried the following code, but it gave an error. > Can someone point me in the right direction? > > ================ > require ‘net/http’ > Net::HTTP.version\_1\_1 # declear to use 1.1 features. > > url0 = “[http://xyz.mycompany.com](http://xyz.mycompany.com)” > url = “[http://xyz.mycompany.com:3000/](http://xyz.mycompany.com:3000/)” > auth = “/scripts/isynch.dll” > account = “me” > passwd = “notmyrealpassword” > > # Net::HTTP.start( url ) {|http| response, body = http.get( auth, ‘Authorization’ =\> 'Basic ’ + [“#{account}:#{passwd}”].pack(‘m’).strip ) print body } Oops, forgot to send the error message: ./get.rb /usr/local/lib/ruby/1.6/net/protocol.rb:469:in `new': getaddrinfo: no address associated with hostname. (SocketError) from /usr/local/lib/ruby/1.6/net/protocol.rb:469:in `connect’ from /usr/local/lib/ruby/1.6/net/protocol.rb:468:in `timeout' from /usr/local/lib/ruby/1.6/net/protocol.rb:468:in `connect’ from /usr/local/lib/ruby/1.6/net/protocol.rb:462:in `initialize' from /usr/local/lib/ruby/1.6/net/protocol.rb:159:in `new’ from /usr/local/lib/ruby/1.6/net/protocol.rb:159:in `conn_socket' from /usr/local/lib/ruby/1.6/net/protocol.rb:148:in `connect’ from /usr/local/lib/ruby/1.6/net/protocol.rb:142:in `_start' from /usr/local/lib/ruby/1.6/net/protocol.rb:128:in `start’ from /usr/local/lib/ruby/1.6/net/http.rb:455:in `start’ from ./get.rb:14 > **···** > > On Thu, Sep 26, 2002 at 03:20:19AM +0900, Jim Freeze wrote: > > > Thanks > > > > Jim > > ## – Jim Freeze > > Programming Ruby > def initialize; fun; end > A language with class ---
**Author:** ![GOTO\_Kentaro1](https://avatars.discourse-cdn.com/v4/letter/g/e5b9ba/32.png) [@GOTO\_Kentaro1](https://rubytalk.org/u/GOTO_Kentaro1)\ **Post date:** [25 September 2002 19:47 UTC](https://rubytalk.org/t/http-net/1982/6 "2002-09-25T19:47:51Z")
> > Net::HTTP.start( url ) {|http| should be Net::HTTP.start(host[, port]){|http| > ./get.rb > /usr/local/lib/ruby/1.6/net/protocol.rb:469:in `new’: getaddrinfo: no address associated with hostname. (SocketError) this was caused when Net::HTTP would lookup address of host “http://…”. Because http://… is not a hostname and DNS lookup failed. net/http.rb include reference and samples. Try: % rd2 http.rb \> http.html or % ruby -ne ‘print if /^=begin/…/^/=end/’ http.rb | more – Gotoken > **···** > > At Thu, 26 Sep 2002 04:10:34 +0900, Jim Freeze wrote: ---
**Author:** ![Jim\_Freeze2](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@Jim\_Freeze2](https://rubytalk.org/u/Jim_Freeze2)\ **Post date:** [25 September 2002 20:47 UTC](https://rubytalk.org/t/http-net/1982/7 "2002-09-25T20:47:19Z")
Thanks, that helped. I changed the code to the following: > **···** > > On Thu, Sep 26, 2002 at 04:47:51AM +0900, GOTO Kentaro wrote: > > > At Thu, 26 Sep 2002 04:10:34 +0900, \> Jim Freeze wrote: > > > > > > Net::HTTP.start( url ) {|http| > > > > should be Net::HTTP.start(host[, port]){|http| > > > > > ./get.rb > > > /usr/local/lib/ruby/1.6/net/protocol.rb:469:in `new’: getaddrinfo: no address associated with hostname. (SocketError) > > > > this was caused when Net::HTTP would lookup address of host “http://…”. > > Because http://… is not a hostname and DNS lookup failed. > > > > net/http.rb include reference and samples. Try: > > > > % rd2 http.rb \> http.html > > > > or > > > > % ruby -ne ‘print if /^=begin/…/^/=end/’ http.rb | more > > > > – Gotoken > > ================================================ > require ‘net/http’ > Net::HTTP.version\_1\_1 # declear to use 1.1 features. > > url = “[xyz.mycompany.com](http://xyz.mycompany.com)” > port = 3000 > path = “/scripts/isynch.dll” > account = “me” > passwd = “notarealpassword” > > # Net::HTTP.start( url, port ) {|http| begin response, body = http.get( path, ‘Authorization’ =\> 'Basic ’ + [“#{account}:#{passwd}”].pack(‘m’).strip ) print body rescue Net::ProtoRetriableError =\> err if m = %rhttp://([^/]+).match( err.response[‘location’] ) host = m[1].strip path = m.post\_match retry end end } > > The code works, but I get back a frame: > > Modeling Request System > > Is there way to automatically get the frame documents, or do > I have to search the returned text and filter out the > new paths? > > Thanks > > ## – Jim Freeze > > Programming Ruby > def initialize; fun; end > A language with class ---
**Author:** ![GOTO\_Kentaro1](https://avatars.discourse-cdn.com/v4/letter/g/e5b9ba/32.png) [@GOTO\_Kentaro1](https://rubytalk.org/u/GOTO_Kentaro1)\ **Post date:** [25 September 2002 21:18 UTC](https://rubytalk.org/t/http-net/1982/8 "2002-09-25T21:18:24Z")
You can pick up values of SRC from html as follows: html = … require “uri” current\_uri = URI.parse(“[http://xyz.mycompany.com:3000/scripts/isynch.dll](http://xyz.mycompany.com:3000/scripts/isynch.dll)”) pattern = /|\<.+?\>|[^\<\>]+/m href = /href\s\*=\s\*“(.+?)”/im frame = html.scan(pattern).grep(/^\<\s\*frame\s+.\*src=/i).map{|src| /src=“(.+?)”|src=([^\s\>]+)/i.match(src) current\_uri.merge($1 || $2) } p frame – Gotoken > **···** > > At Thu, 26 Sep 2002 05:47:19 +0900, Jim Freeze wrote: > > > Is there way to automatically get the frame documents, or do > > I have to search the returned text and filter out the > > new paths? ---
**Author:** ![Jim\_Freeze2](https://avatars.discourse-cdn.com/v4/letter/j/e480ec/32.png) [@Jim\_Freeze2](https://rubytalk.org/u/Jim_Freeze2)\ **Post date:** [26 September 2002 02:09 UTC](https://rubytalk.org/t/http-net/1982/9 "2002-09-26T02:09:14Z")
Ok, thanks. Now frame has the value: [#\, #\, #\] …but, sorry to say, I am still a big dummy with regards to net/http. Is there a way to directly use a uri to get the page data? Thanks > **···** > > On Thu, Sep 26, 2002 at 06:18:24AM +0900, GOTO Kentaro wrote: > > > At Thu, 26 Sep 2002 05:47:19 +0900, \> Jim Freeze wrote: > > > > > Is there way to automatically get the frame documents, or do > > > I have to search the returned text and filter out the > > > new paths? > > > > You can pick up values of SRC from html as follows: > > > > html = … > > > > require “uri” > > current\_uri = URI.parse(“[http://xyz.mycompany.com:3000/scripts/isynch.dll](http://xyz.mycompany.com:3000/scripts/isynch.dll)”) > > > > pattern = /|\<.+?\>|[^\<\>]+/m > > href = /href\s\*=\s\*“(.+?)”/im > > frame = html.scan(pattern).grep(/^\<\s\*frame\s+.\*src=/i).map{|src| > > /src=“(.+?)”|src=([^\s\>]+)/i.match(src) > > current\_uri.merge($1 || $2) > > } > > p frame > > > > – Gotoken > > ## – Jim Freeze > > Programming Ruby > def initialize; fun; end > A language with class ---
**Author:** ![Minero\_Aoki1](https://avatars.discourse-cdn.com/v4/letter/m/fbc32d/32.png) [@Minero\_Aoki1](https://rubytalk.org/u/Minero_Aoki1)\ **Post date:** [26 September 2002 02:28 UTC](https://rubytalk.org/t/http-net/1982/10 "2002-09-26T02:28:37Z")
Hi, In mail “Re: http/net” > …but, sorry to say, I am still a big dummy with regards to net/http. > Is there a way to directly use a uri to get the page data? Net::HTTP.get accepts URI objects. Other Net::HTTP methods do NOT accept URI objects. Because: Net::HTTP.start(‘[www.example.com](http://www.example.com)’) {|http| print http.get(URI.parse(‘[http://www.ruby-lang.org/pub/ruby-1.6.6.tar.gz](http://www.ruby-lang.org/pub/ruby-1.6.6.tar.gz)’)) } This operation does not make sense. – Minero Aoki > **···** > > Jim Freeze [jim@freeze.org](mailto:jim@freeze.org) wrote: ---
**Author:** ![Minero\_Aoki1](https://avatars.discourse-cdn.com/v4/letter/m/fbc32d/32.png) [@Minero\_Aoki1](https://rubytalk.org/u/Minero_Aoki1)\ **Post date:** [26 September 2002 02:34 UTC](https://rubytalk.org/t/http-net/1982/11 "2002-09-26T02:34:54Z")
In mail “Re: http/net” > **···** > > Minero Aoki [aamine@mx.edit.ne.jp](mailto:aamine@mx.edit.ne.jp) wrote: > > > Net::HTTP.get accepts URI objects. > > This change is done _ **in 1.7** _. > > – Minero Aoki.