TCP/IP protocol and Net::HTTP

ruby-talk
1

Works for me too.

You aren’t being stung by some institutional proxy server or something
like that (sorry I don’t know much about networks) are you? I imagine
ad.doubleclick.net gets a lot of hits.

Jon

···

-----Original Message-----
From: Nigel Gilbert [mailto:n.gilbert@soc.surrey.ac.uk]
Sent: 18 June 2003 19:25
To: ruby-talk ML
Subject: TCP/IP protocol and Net::HTTP

For almost all web sites, such as www.ruby-lang.org, the code

Net::HTTP.get_print ‘www.ruby-lang.org’, ‘/’

works without a problem, returning the home page.

However, ‘ad.doubleclick.net’ is an exception. The code;

Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’

times out. If you access this site in a web browser, there is no
problem getting the home page from doubleclick; it arrives almost
immediately.

By watching the ethernet traffic, it seems that doubleclick replies
from a different IP address than the address corresponding to the
domain name, but ruby keeps listening to responses from the domain
address (note this is not a 302 redirect; it is all happening at a
much lower level). Eventually, there is a timeout error from
/usr/local/lib/ruby/1.6/net/protocol.rb:449:in `initialize’. Is this
possible within the TCP/IP protocol? Is this something that has been
dealt with in a latter release (I am still using ruby 1.6.8). Is there
some workaround?

Nigel

This e-mail has been scanned for all viruses by Star Internet. The
service is powered by MessageLabs. For more information on a proactive
anti-virus service working around the clock, around the globe, visit:

This message has been checked for all known viruses by Star Internet delivered
through the MessageLabs Virus Control Centre. For further information visit

Any views or personal opinions expressed within this email may not be those of Talis Information Ltd.
The content of this email message and any files that may be attached are confidential, and for the usage of the intended recipient only. If you are not the intended recipient, then please return this message to the sender and delete it. Any use of this e-mail by an unauthorised recipient is prohibited.

2

I am beginning to wonder if this problem arises from the MacOS X
implementation of ruby. Those who have tried to reproduce the problem
(thanks to you all) have had no difficulty accessing doubleclick from
Linux and I have also succeeded with a Solaris implementation at my
university (could someone try it on Windows?)

On the other hand, I get the timeout error on MacOS X both at home
(with a cable modem connection) and at my university (with a direct
connection to the internet).

If there is someone out there using MacOSX, could you try it and see
whether this theory is correct? What you need to do is:

  1. fire up irb
  2. enter
    require ‘net/http’
  3. enter
    Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’
  4. tell me whether you get a Timeout error, or alternatively some lines
    of HTML, starting:
This is a DoubleClick advertising server.

Thanks.

Nigel

···

On Thursday, June 19, 2003, at 11:23 AM, J.Hawkesworth wrote:

Works for me too.

You aren’t being stung by some institutional proxy server or something
like that (sorry I don’t know much about networks) are you? I imagine
ad.doubleclick.net gets a lot of hits.

Jon

-----Original Message-----
From: Nigel Gilbert [mailto:n.gilbert@soc.surrey.ac.uk]
Sent: 18 June 2003 19:25
To: ruby-talk ML
Subject: TCP/IP protocol and Net::HTTP

For almost all web sites, such as www.ruby-lang.org, the code

Net::HTTP.get_print ‘www.ruby-lang.org’, ‘/’

works without a problem, returning the home page.

However, ‘ad.doubleclick.net’ is an exception. The code;

Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’

times out. If you access this site in a web browser, there is no
problem getting the home page from doubleclick; it arrives almost
immediately.

By watching the ethernet traffic, it seems that doubleclick replies
from a different IP address than the address corresponding to the
domain name, but ruby keeps listening to responses from the domain
address (note this is not a 302 redirect; it is all happening at a
much lower level). Eventually, there is a timeout error from
/usr/local/lib/ruby/1.6/net/protocol.rb:449:in `initialize’. Is this
possible within the TCP/IP protocol? Is this something that has been
dealt with in a latter release (I am still using ruby 1.6.8). Is there
some workaround?

Nigel

_
This e-mail has been scanned for all viruses by Star Internet. The
service is powered by MessageLabs. For more information on a proactive
anti-virus service working around the clock, around the globe, visit:
http://www.star.net.uk

_

This message has been checked for all known viruses by Star Internet
delivered
through the MessageLabs Virus Control Centre. For further information
visit
http://www.star.net.uk/stats.asp

Any views or personal opinions expressed within this email may not be
those of Talis Information Ltd.
The content of this email message and any files that may be attached
are confidential, and for the usage of the intended recipient only. If
you are not the intended recipient, then please return this message to
the sender and delete it. Any use of this e-mail by an unauthorised
recipient is prohibited.

3

I get this with ruby 1.7.3 (2002-10-12) [i386-mswin32] on Win 2000:

irb(main):001:0> require ‘net/http’
true
irb(main):002:0> Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’

This is a DoubleClick advertising server. ... more lines ... nil irb(main):003:0>

Regards,
Pit

···

On 19 Jun 2003 at 21:08, Nigel Gilbert wrote:

I am beginning to wonder if this problem arises from the MacOS X
implementation of ruby. Those who have tried to reproduce the problem
(thanks to you all) have had no difficulty accessing doubleclick from
Linux and I have also succeeded with a Solaris implementation at my
university (could someone try it on Windows?)

4

Mixed results for me on 10.2.6 with ruby 1.6.7 (2002-03-01)
[powerpc-darwin6.0] - sometimes it works and sometimes it times out.
Even when it works it still very slow.

Dave.

···

On Thursday, June 19, 2003, at 01:08 pm, Nigel Gilbert wrote:

I am beginning to wonder if this problem arises from the MacOS X
implementation of ruby. Those who have tried to reproduce the problem
(thanks to you all) have had no difficulty accessing doubleclick from
Linux and I have also succeeded with a Solaris implementation at my
university (could someone try it on Windows?)

On the other hand, I get the timeout error on MacOS X both at home
(with a cable modem connection) and at my university (with a direct
connection to the internet).

If there is someone out there using MacOSX, could you try it and see
whether this theory is correct? What you need to do is:

  1. fire up irb
  2. enter
    require ‘net/http’
  3. enter
    Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’
  4. tell me whether you get a Timeout error, or alternatively some
    lines of HTML, starting:
This is a DoubleClick advertising server.

Thanks.

Nigel

On Thursday, June 19, 2003, at 11:23 AM, J.Hawkesworth wrote:

Works for me too.

You aren’t being stung by some institutional proxy server or something
like that (sorry I don’t know much about networks) are you? I imagine
ad.doubleclick.net gets a lot of hits.

Jon

-----Original Message-----
From: Nigel Gilbert [mailto:n.gilbert@soc.surrey.ac.uk]
Sent: 18 June 2003 19:25
To: ruby-talk ML
Subject: TCP/IP protocol and Net::HTTP

For almost all web sites, such as www.ruby-lang.org, the code

Net::HTTP.get_print ‘www.ruby-lang.org’, ‘/’

works without a problem, returning the home page.

However, ‘ad.doubleclick.net’ is an exception. The code;

Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’

times out. If you access this site in a web browser, there is no
problem getting the home page from doubleclick; it arrives almost
immediately.

By watching the ethernet traffic, it seems that doubleclick replies
from a different IP address than the address corresponding to the
domain name, but ruby keeps listening to responses from the domain
address (note this is not a 302 redirect; it is all happening at a
much lower level). Eventually, there is a timeout error from
/usr/local/lib/ruby/1.6/net/protocol.rb:449:in `initialize’. Is this
possible within the TCP/IP protocol? Is this something that has been
dealt with in a latter release (I am still using ruby 1.6.8). Is
there
some workaround?

Nigel

__
This e-mail has been scanned for all viruses by Star Internet. The
service is powered by MessageLabs. For more information on a proactive
anti-virus service working around the clock, around the globe, visit:
http://www.star.net.uk

__

This message has been checked for all known viruses by Star Internet
delivered
through the MessageLabs Virus Control Centre. For further information
visit
http://www.star.net.uk/stats.asp

Any views or personal opinions expressed within this email may not be
those of Talis Information Ltd.
The content of this email message and any files that may be attached
are confidential, and for the usage of the intended recipient only.
If you are not the intended recipient, then please return this
message to the sender and delete it. Any use of this e-mail by an
unauthorised recipient is prohibited.

5

Hi,

I am beginning to wonder if this problem arises from the MacOS X
implementation of ruby.
[…]
On the other hand, I get the timeout error on MacOS X both at home
(with a cable modem connection) and at my university (with a direct
connection to the internet).

Hi, I get the same timeout on OS X (over a DSL business line)
with: ruby 1.6.7 (2002-03-01) [powerpc-darwin6.3]

I ran a tcpdump when executing the
Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’
line and was surprised to see NO tcp packets transmitted to
ad.doubleclick.net at any time.

I ran a ktrace - but it’s my first time using it and I’m not
sure how to interpret what I’m seeing… :slight_smile: Why is the SIGALRM
happening before the socket calls? The call listed as taking
all the time is the “51.224883 CALL socket(0x2,0x1,0x6)”…

23026 ruby 0.001869 PSIG SIGVTALRM caught handler=0x1a0334 mask=0x0 code=0x0
23026 ruby 0.000121 CALL sigreturn(0xbffcc660)
23026 ruby 0.000047 RET sigreturn JUSTRETURN
23026 ruby 0.003751 CALL gettimeofday(0xbffbc350,0)
23026 ruby 0.000130 RET gettimeofday 1056027683/0x3ef1b423
23026 ruby 0.000040 CALL gettimeofday(0xbffbc2b0,0)
23026 ruby 0.000030 RET gettimeofday 1056027683/0x3ef1b423
23026 ruby 0.000032 CALL gettimeofday(0xbffbbfe0,0)
23026 ruby 0.000030 RET gettimeofday 1056027683/0x3ef1b423
23026 ruby 0.000029 CALL select(0,0xbffbc050,0xbffbc0d0,0xbffbc150,0xbffbc1d0)
23026 ruby 0.000036 RET select 0
23026 ruby 51.224883 CALL socket(0x2,0x1,0x6)
23026 ruby 0.000232 RET socket 7
23026 ruby 0.000042 CALL fcntl(0x7,0x3,0)
23026 ruby 0.000031 RET fcntl 2
23026 ruby 0.000071 CALL fcntl(0x7,0x4,0x6)
23026 ruby 0.000044 RET fcntl 0
23026 ruby 0.000030 CALL connect(0x7,0x47eba0,0x10)
23026 ruby 0.000120 RET connect -1 errno 36 Operation now in progress
23026 ruby 0.000056 CALL gettimeofday(0xbffbe760,0)
23026 ruby 0.000033 RET gettimeofday 1056027734/0x3ef1b456
23026 ruby 0.000029 CALL select(0x9,0xbffbe7d0,0xbffbe850,0xbffbe8d0,0xbffbe950)
23026 ruby 0.000038 RET select 0
23026 ruby 0.000974 CALL gettimeofday(0xbffbc350,0)
23026 ruby 0.000223 RET gettimeofday 1056027734/0x3ef1b456
23026 ruby 0.001292 CALL sigprocmask(0x3,0x2178a4,0)
23026 ruby 0.000132 RET sigprocmask 0
23026 ruby 0.001001 CALL sigprocmask(0x3,0x2178a4,0)
23026 ruby 0.000097 RET sigprocmask 0
23026 ruby 0.000107 CALL write(0x1,0x342000,0xc)
23026 ruby 0.000081 GIO fd 1 wrote 12 bytes
“TimeoutError”

…anyway… On a lark I became curious about DNS lookup time
and tried from the shell:

$ host ad.doubleclick.net

…and it took an awful long time to come back under OS X.

So from ruby I tried the literal IP addr for ad.doubleclick.net:

irb(main):007:0> Net::HTTP.get_print ‘216.73.86.70’, ‘/’

This is a DoubleClick advertising server.

Click here for the DoubleClick home page.

Click here for privacy information. nil

The above is under OS X…

So it seems maybe some DNS related thing under OS X and not
Ruby specifically… (?)

Hope this helps,

Bill

···

From: “Nigel Gilbert” n.gilbert@soc.surrey.ac.uk

6

Nigel Gilbert n.gilbert@soc.surrey.ac.uk writes:

If there is someone out there using MacOSX, could you try it and see
whether this theory is correct? What you need to do is:

I just tried under OS X and see the same timeout problem.

~> irb
irb(main):001:0> require ‘net/http’
=> true
irb(main):002:0> Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’
TimeoutError: execution expired
from /usr/local/lib/ruby/1.6/timeout.rb:37:in connect' from /usr/local/lib/ruby/1.6/net/protocol.rb:467:in timeout’
[snip]

Jim

···

–
Jim Menard, jimm@io.com, http://www.io.com/~jimm/
“Slideshow … boring. Loosing … consciousness …” – The Tick

7

Both our systems:

  • Mac OS X 10.2.6 / ruby 1.6.7 (2002-03-01) [powerpc-darwin6.0]
  • Red Hat Linux 8.0 / ruby 1.6.8 (2002-12-24) [i686-linux]

fail this with a timeout. We have a crummy satellite connection through
DirecTV, but then performing the same request in the web browser yields
the page you mentioned.

···

On Thursday, Jun 19, 2003, at 07:08 America/Chicago, Nigel Gilbert wrote:

I am beginning to wonder if this problem arises from the MacOS X
implementation of ruby. Those who have tried to reproduce the problem
(thanks to you all) have had no difficulty accessing doubleclick from
Linux and I have also succeeded with a Solaris implementation at my
university (could someone try it on Windows?)

On the other hand, I get the timeout error on MacOS X both at home
(with a cable modem connection) and at my university (with a direct
connection to the internet).

If there is someone out there using MacOSX, could you try it and see
whether this theory is correct?..

–
John Platte
Principal Consultant, NIKA Consulting
http://nikaconsulting.com/

8

My results are below. I am using Mac OS X 10.2.6 and Ruby 1.8p2. I also
tried ‘ad.doubleclick.net’ in Safari and got the same page. If you
haven’t already, I recommend trying your browser also to test whether
your problem may be perhaps firewall related.

Regards,

Mark

irb(main):001:0> require ‘net/http’
=> true
irb(main):002:0> Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’

This is a DoubleClick advertising server.

Click here for the DoubleClick home page.

Click here for privacy information. => nil irb(main):003:0>

···

On Thursday, June 19, 2003, at 08:08 AM, Nigel Gilbert wrote:

[snip]

If there is someone out there using MacOSX, could you try it and see
whether this theory is correct? What you need to do is:
[snip]

9

If there is someone out there using MacOSX, could you try it and see
whether this theory is correct? What you need to do is:

I get a timeout too.

[zen:~] koan% ruby -v
ruby 1.6.7 (2002-03-01) [powerpc-darwin6.0]
[zen:~] koan% sw_vers
ProductName: Mac OS X
ProductVersion: 10.2.6
BuildVersion: 6L60
[zen:~] koan% uname -a
Darwin zen.kotnet.org 6.6 Darwin Kernel Version 6.6: Thu May 1
21:48:54 PDT 2003; root:xnu/xnu-344.34.obj~1/RELEASE_PPC Power
Macintosh powerpc

Koen

···

–
“The human mind ordinarily operates at only ten percent of its
capacity; the rest is overhead for the operating system.”

10

So it *seems* maybe some DNS related thing under OS X and not
Ruby specifically..... (?)

See also [ruby-talk:66272]

Guy Decoux

11

As Guy mentioned, this is a doubleclick issue; they’re not responding
to AAAA (ipv6?) requests, which is what Ruby is issuing.

···

— Jim Menard jimm@io.com wrote:

Nigel Gilbert n.gilbert@soc.surrey.ac.uk writes:

If there is someone out there using MacOSX, could you try it and
see
whether this theory is correct? What you need to do is:

I just tried under OS X and see the same timeout problem.

Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!

12

Thanks to all who have run trials and especially to Guy Decoux who
referenced an earlier discussion in ruby-talk that shed light on the
problem (which was: Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’ times
out under MacOSX, but not on other platforms). My summary is:

On MacOSX, the implementation of getaddrinfo (which is used to
translate ‘ad.doubleclick.net’ into an IP address) starts by sending a
query to the doubleclick DNS server using an IPv6 format. This DNS
server should either respond with the desired IP address, or decline to
do so. In fact, the doubleclick DNS server makes no response at all,
and the Mac keeps waiting until eventually it times out.

This explains why the problem only seems to occur only with the
combination of (a) doubleclick [their DNS server is unusual [1] in that
it doesn’t respond) and (b) MacOS X (this implementation of
getaddrinfo first tries an IPv6 format, and then backs down to an IPv4
format query, while other ruby implementations don’t attempt an IPv6
query at all).

The work around (not yet tried) is to rebuild ruby with the configure
options:

       --enable-ipv6 \
       --with-lookup-order-hack=INET

which essentially says try IPv4 before IPv6

The earlier ruby-talk thread related to all this starts at 66245

I hope that the above options could become the default in the next
release.

Nigel

[1] Or perhaps one could say, ‘buggy’

···

On Thursday, June 19, 2003, at 05:33 PM, Michael Campbell wrote:

— Jim Menard jimm@io.com wrote:

Nigel Gilbert n.gilbert@soc.surrey.ac.uk writes:

If there is someone out there using MacOSX, could you try it and
see
whether this theory is correct? What you need to do is:

I just tried under OS X and see the same timeout problem.

As Guy mentioned, this is a doubleclick issue; they’re not responding
to AAAA (ipv6?) requests, which is what Ruby is issuing.

Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com

13

I spoke (wrote?) too soon!

The work around mentioned below does not seem to cure the problem (even
building with --disable-ipv6 has no benefits). Any other ideas?

Nigel

···

On Thursday, June 19, 2003, at 05:57 PM, Nigel Gilbert wrote:

Thanks to all who have run trials and especially to Guy Decoux who
referenced an earlier discussion in ruby-talk that shed light on the
problem (which was: Net::HTTP.get_print ‘ad.doubleclick.net’, ‘/’
times out under MacOSX, but not on other platforms). My summary is:

On MacOSX, the implementation of getaddrinfo (which is used to
translate ‘ad.doubleclick.net’ into an IP address) starts by sending a
query to the doubleclick DNS server using an IPv6 format. This DNS
server should either respond with the desired IP address, or decline
to do so. In fact, the doubleclick DNS server makes no response at
all, and the Mac keeps waiting until eventually it times out.

This explains why the problem only seems to occur only with the
combination of (a) doubleclick [their DNS server is unusual [1] in
that it doesn’t respond) and (b) MacOS X (this implementation of
getaddrinfo first tries an IPv6 format, and then backs down to an IPv4
format query, while other ruby implementations don’t attempt an IPv6
query at all).

The work around (not yet tried) is to rebuild ruby with the configure
options:

      --enable-ipv6 \
      --with-lookup-order-hack=INET

which essentially says try IPv4 before IPv6

The earlier ruby-talk thread related to all this starts at 66245

I hope that the above options could become the default in the next
release.

Nigel

[1] Or perhaps one could say, ‘buggy’

On Thursday, June 19, 2003, at 05:33 PM, Michael Campbell wrote:

— Jim Menard jimm@io.com wrote:

Nigel Gilbert n.gilbert@soc.surrey.ac.uk writes:

If there is someone out there using MacOSX, could you try it and
see
whether this theory is correct? What you need to do is:

I just tried under OS X and see the same timeout problem.

As Guy mentioned, this is a doubleclick issue; they’re not responding
to AAAA (ipv6?) requests, which is what Ruby is issuing.

Do you Yahoo!?
SBC Yahoo! DSL - Now only $29.95 per month!
http://sbc.yahoo.com