Hi all,
I am building a nice little system using Ruby (on Rails), one part of
which uses Net::HTTP to retrieve some data over HTTP. Everything seems
to work fine, but on some requests, I get an EOFError.
As I found out, this problem has already been reported, but without any
answer. See:
http://rubyforge.org/forum/forum.php?thread_id=28826&forum_id=6052
I think I may have traced the problem back to a bug in Net::HTTP. Here
is a two-liner to reproduce the error:
$ irb
>> require 'net/http'
=> true
>> res = Net::HTTP.get_response(URI.parse('http://snapcasa.com/get.aspx?code=1000&size=m&url=www.google.com'))
EOFError: end of file reached
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:133:in `sysread'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:133:in `rbuf_fill'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:56:in `timeout'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/timeout.rb:76:in `timeout'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:132:in `rbuf_fill'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:116:in `readuntil'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/protocol.rb:126:in `readline'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2236:in `read_chunked'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2216:in `read_body_0'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2182:in `read_body'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2207:in `body'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:2146:in `reading_body'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:1061:in `request'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:957:in `request_get'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:380:in `get_response'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:547:in `start'
from /System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/net/http.rb:379:in `get_response'
from (irb):2>>
My quick analysis:
In http.rb, line 2236 the "read_chunked" function calls
@socket.readline. This "readline" function, in protocol.rb line 126,
calls readuntil("\n"). This works fine if the data chunk is
"\n-terminated", but throws an EOFError if it is not.
I am not sure about the underlying standards of chunked http, maybe the
data chunk is supposed to always been \n-terminated, and it may be a
mis-behaving server, but the fact is: I can get the example image fine
with any browser, but not with Net::HTTP.
As a tentative fix, I wrote a patch that catches the EOFError in
read_chunked. You will the patch file attached. With the patch, things
work fine:
$ irb
>> require 'net/http'
=> true
>> res = Net::HTTP.get_response(URI.parse('http://snapcasa.com/get.aspx?code=1000&size=m&url=www.google.com'))
=> #<Net::HTTPOK 200 OK readbody=true>
>> File.open('test.jpg','w').write res.body
=> 2881
>>
With the patch, the above gives me a perfectly fine JPEG file. However,
I am afraid my current patch, with a big begin...rescue around most of
the body of the read_chunked function, catches the EOFError at level
higher than necessary, which is not good practice...
Anyway, before continuing any further, could someone involved in the
development of Ruby take a look at this, confirm the existence of the
bug, and maybe even come up with a better fix?
PS: please let me know if I posted this in the wrong list, or if I
should open a bug report on some bug tracking system.
Thank you for your help.
read_chunked_eoferror_tentative_fix.patch (1.05 KB)
···
--
Yves-Eric