Reading an HTTPResponse line by line

ruby-talk
1

Hi,
how can I read a response from an http server line by line?

req = Net::HTTP.new('10.2.1.2')
res = req.post('/mycgi', mydata) do |str|
         print str
       end

How can I read a line instead of a "chunck" of data in str?
Something like "BufferedReader.readLine()" in Java...

2

harp:~ > cat a.rb
   require 'stringio'

   sio = StringIO.new
   req = Net::HTTP.new '10.2.1.2'
   mydata = ''
   res = req.post('/mycgi', mydata){|str| sio << str}
   sio.rewind
   sio.each{|line| p line}

-a

···

On Fri, 19 Jan 2007, Scara Maccai wrote:

Hi,
how can I read a response from an http server line by line?

req = Net::HTTP.new('10.2.1.2')
res = req.post('/mycgi', mydata) do |str|
       print str
     end

How can I read a line instead of a "chunck" of data in str?
Something like "BufferedReader.readLine()" in Java...

--
in the practice of tolerance, one's enemy is the best teacher.
- the dalai lama

3

Scara Maccai wrote:

Hi,
how can I read a response from an http server line by line?

req = Net::HTTP.new('10.2.1.2')
res = req.post('/mycgi', mydata) do |str|
         print str
       end

How can I read a line instead of a "chunck" of data in str?
Something like "BufferedReader.readLine()" in Java...

Do you mean something like:

require 'net/http'

h = Net::HTTP.new('www.ruby-lang.org', 80)
resp, data = h.get('/index.html', nil)

data.each_line do |line|
  puts "--> #{line}"
end

4

In every case (your way or the StringIO way) it looks to me that the response is parsed at the very end, so I have to wait for the whole response to be finished before starting using it: with "BufferedReader.readLine()" I use the lines as soon as they are available. Is there a way to do that with Ruby?

(thank you both for your answers)

···

Do you mean something like:

require 'net/http'

h = Net::HTTP.new('www.ruby-lang.org', 80)
resp, data = h.get('/index.html', nil)

data.each_line do |line|
  puts "--> #{line}"
end

5

just give the post method a block. for instance, although google doesn't
allow posting to this url

   harp:~ > cat a.rb
   require 'net/http'

   Net::HTTP.start 'fortytwo.merseine.nu', 80 do |http|
     http.post '/form.cgi', 'k=v&a=42' do |chunk|
       puts chunk
     end
   end

   harp:~ > ruby a.rb

···

On Fri, 19 Jan 2007, Scara Maccai wrote:

In every case (your way or the StringIO way) it looks to me that the response is parsed at the very end, so I have to wait for the whole response to be finished before starting using it: with "BufferedReader.readLine()" I use the lines as soon as they are available. Is there a way to do that with Ruby?

   ---
   k:
   - v
   a:
   - "42"

-a
--
in the practice of tolerance, one's enemy is the best teacher.
- the dalai lama

6

Ok, but that's what I did in the first place, but it doesn't allow reading line by line... that is, it looks to me that you can

1) reading as chunks come in

OR

2) reading line by line

but you can't

"reading lines as they come in..."

(that is, 1) AND 2) as it would be the case with "BufferedReader.readLine()" in Java).

In other words what I wanted is (pseudo-code):

response_handler = get_response_handler
while( line = response_handler.getLine())
{
  doSomethingWith(line)
}

without having to wait for the whole response to be read before starting parsing...

···

just give the post method a block. for instance, although google doesn't
allow posting to this url

   harp:~ > cat a.rb
   require 'net/http'

   Net::HTTP.start 'fortytwo.merseine.nu', 80 do |http|
     http.post '/form.cgi', 'k=v&a=42' do |chunk|
       puts chunk
     end
   end

   harp:~ > ruby a.rb
   ---
   k:
   - v
   a:
   - "42"

-a

7

Scara Maccai wrote:

Ok, but that's what I did in the first place, but it doesn't allow reading line by line... that is, it looks to me that you can

1) reading as chunks come in

OR

2) reading line by line

but you can't

"reading lines as they come in..."

chunks are made out of lines; it's not so difficult to process each
line as the chunks come in:

buffer = ""
req = Net::HTTP.new('10.2.1.2')
req.post('/mycgi', mydata) do |chunk|
   buffer << chunk
   while buffer.sub!(/^(.*)\n/,"")
     do_something_with_line($1)
   end
end

Daniel

8

Ok, that's what I was looking for!
Thank you everybody

Daniel DeLorme wrote:

···

Scara Maccai wrote:

but you can't

"reading lines as they come in..."

chunks are made out of lines; it's not so difficult to process each
line as the chunks come in:

buffer = ""
req = Net::HTTP.new('10.2.1.2')
req.post('/mycgi', mydata) do |chunk|
   buffer << chunk
   while buffer.sub!(/^(.*)\n/,"")
     do_something_with_line($1)
   end
end

Daniel