TCPSocket.gethostbyname difficulties

ruby-talk
1

I’m trying to use TCPSocket.gethostbyname to verify that a given domain
actually exists in DNS. For some reason, though, some domains fail to
resolve that resolve fine using other applications. For instance:

irb(main):001:0> require ‘socket’
=> true
irb(main):002:0> TCPSocket.gethostbyname(‘noblepack.com’)
SocketError: host not found
from (irb):2:in `gethostbyname’
from (irb):2
irb(main):003:0> TCPSocket.gethostbyname(‘google.com’)
=> [“www.google.com”, [], 2, “216.239.37.100”]

I can browse to either of those hosts, so what’s different about them? Any
help would be greatly appreciated; my guess is that I just fail to
understand something critical about DNS and/or gethostbyname.

Thanks,

Nathaniel

<:((><

2

I can browse to either of those hosts, so what’s different about them? Any
help would be greatly appreciated; my guess is that I just fail to
understand something critical about DNS and/or gethostbyname.

I just used dnstracer to find the DNS server that knows all about those
hosts. Apparently google.com is stored at the DNS server my machine got
from the DHCP server. noblepack.com however is not stored at that DNS
server, but just like any good DNS server, it knows at which DNS server it
is stored (DNS is hierarchical, that’s how it knows). So the lookup
process possibly needs to follow a chain of DNS servers, and I’m sure this
is done by all browsers, but maybe not by ruby (although I’d expect that
to be done by the OS or its libraries…)

Seems also that sometimes dnstracer tells me the hostname is known by
another DNS server, but still ruby can find it. My guess is that’s because
of caching at DNS servers. The reverse however seems always to be true: if
ruby can’t find the hostname, it’s not stored at my local DNS server…

Just a guess that seems to fit the symptoms…

Peter

3

Hi,

···

In message “TCPSocket.gethostbyname difficulties” on 03/09/26, “Nathaniel Talbott” nathaniel@NOSPAMtalbott.ws writes:

I’m trying to use TCPSocket.gethostbyname to verify that a given domain
actually exists in DNS. For some reason, though, some domains fail to
resolve that resolve fine using other applications. For instance:

irb(main):001:0> require ‘socket’
=> true
irb(main):002:0> TCPSocket.gethostbyname(‘noblepack.com’)
SocketError: host not found
from (irb):2:in `gethostbyname’
from (irb):2

This works for me:

ruby -r socket -e ‘p TCPSocket.gethostbyname(“noblepack.com”)’
ruby 1.8.0 (2003-09-27) [i686-linux]
[“noblepack.com”, , 2, “205.178.141.161”]

Could you tell us more info?

						matz.
4

Hmmmm… I thought TCPSocket.gethostbyname was basically a direct wrapper
for the gethostbyname C library function. It really surprises me that it
doesn’t automatically follow the DNS servers itself.

Does anyone know how to get this to work? Even if I have to add more of my
own code, I need to be able to check if a given hostname is valid or not.

Thanks,

Nathaniel

<:((><

···

Peter [mailto:Peter.Vanbroekhoven@cs.kuleuven.ac.be] wrote:

So the lookup
process possibly needs to follow a chain of DNS servers, and
I’m sure this is done by all browsers, but maybe not by ruby
(although I’d expect that to be done by the OS or its libraries…)

5

Yukihiro Matsumoto wrote:

This works for me:

ruby -r socket -e ‘p TCPSocket.gethostbyname(“noblepack.com”)’
ruby 1.8.0 (2003-09-27) [i686-linux]
[“noblepack.com”, , 2, “205.178.141.161”]

Could you tell us more info?

Out of curiosity, I just tried the above, on my Win2k laptop, running
/\ndy’s 1.8 build:

C:>ruby -v
ruby 1.8.0 (2003-08-04) [i386-mswin32]

C:> ruby -r socket -e ‘p TCPSocket.gethostbyname(“noblepack.com”)’
-e:1:in `gethostbyname’: host not found (SocketError)
from -e:1

However, I can ping that location just fine.

I then tried a different address:

C:> ruby -r socket -e ‘p TCPSocket.gethostbyname(“jamesbritt.com”)’
[“amsnac3.com”, , 2, “66.246.52.127”]

James Britt

6

In article 1064759167.506592.9057.nullmailer@picachu.netlab.jp,

Hi,

I’m trying to use TCPSocket.gethostbyname to verify that a given domain
actually exists in DNS. For some reason, though, some domains fail to
resolve that resolve fine using other applications. For instance:

irb(main):001:0> require ‘socket’
=> true
irb(main):002:0> TCPSocket.gethostbyname(‘noblepack.com’)
SocketError: host not found
from (irb):2:in `gethostbyname’
from (irb):2

This works for me:

ruby -r socket -e ‘p TCPSocket.gethostbyname(“noblepack.com”)’
ruby 1.8.0 (2003-09-27) [i686-linux]
[“noblepack.com”, , 2, “205.178.141.161”]

Could you tell us more info?

Although not addressed to me, on Mandrake Linux 9.1 I see

[mike@ratdog mike]$ ruby -v -r socket -e ‘p TCPSocket.gethostbyname(“noblepack.com”)’
ruby 1.8.0 (2003-09-27) [i686-linux]
-e:1:in `gethostbyname’: host not found (SocketError)
from -e:1
[mike@ratdog mike]$ ping -c 1 noblepack.com
PING noblepack.com (205.178.141.161) 56(84) bytes of data.
64 bytes from 205.178.141.161: icmp_seq=1 ttl=119 time=52.9 ms

— noblepack.com ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 52.996/52.996/52.996/0.000 ms
[mike@ratdog mike]$ ruby -v -r socket -e ‘p TCPSocket.gethostbyname(“stok.co.uk”)’
ruby 1.8.0 (2003-09-27) [i686-linux]
[“berke-breathed.deathtongue.org”, , 2, “209.251.75.53”]

Hope this helps,

Mike

···

Yukihiro Matsumoto matz@ruby-lang.org wrote:

In message “TCPSocket.gethostbyname difficulties” > on 03/09/26, “Nathaniel Talbott” nathaniel@NOSPAMtalbott.ws writes:
–
mike@stok.co.uk | The “`Stok’ disclaimers” apply.
http://www.stok.co.uk/~mike/ | GPG PGP Key 1024D/059913DA
mike@exegenix.com | Fingerprint 0570 71CD 6790 7C28 3D60
http://www.exegenix.com/ | 75D2 9EC4 C1C0 0599 13DA

7

[…]

This works for me:

ruby -r socket -e ‘p TCPSocket.gethostbyname(“noblepack.com”)’
ruby 1.8.0 (2003-09-27) [i686-linux]
[“noblepack.com”, , 2, “205.178.141.161”]

Could you tell us more info?

This surprises me, since there is no PTR record for 205.178.141.161,
and ruby (1.8.0 in my case) insists on doing a gethostbyaddr() on
that address (in sock_hostbyname()) and will raise a “host not
found” error if gethostbyaddr() fails (usually if there is no PTR
record). That lookup is performed regardless of the value of
Socket.do_not_reverse_lookup.

Incidentally, what is the rationale behind requiring the presence of
a PTR record? There are many hosts that do not have one, especially
virtual web hosts, and I have found Socket/TCPSocket.gethostbyname to be
fairly useless as a result (and instead resorted to using
Resolv.getaddress[es]).

		Reimer Behrends
···

Yukihiro Matsumoto (matz@ruby-lang.org) wrote:

8

Does anyone know how to get this to work?

Ask them to correctly register the address, it's a .COM no ?

svg% host noblepack.com
noblepack.com has address 205.178.141.161
svg%

svg% host 205.178.141.161
Host 161.141.178.205.in-addr.arpa not found: 3(NXDOMAIN)
svg%

Guy Decoux

9

Hi,

···

In message “Re: TCPSocket.gethostbyname difficulties” on 03/09/29, James Britt jamesUNDERBARb@seemyemail.com writes:

Out of curiosity, I just tried the above, on my Win2k laptop, running
/\ndy’s 1.8 build:

C:>ruby -v
ruby 1.8.0 (2003-08-04) [i386-mswin32]

C:> ruby -r socket -e ‘p TCPSocket.gethostbyname(“noblepack.com”)’
-e:1:in `gethostbyname’: host not found (SocketError)
from -e:1

However, I can ping that location just fine.

Hmm, might be Win32 specific problem. Does stopping reverse lookup
make any difference?

Socket.do_not_reverse_lookup = false

						matz.
10

Hi,

···

In message “Re: TCPSocket.gethostbyname difficulties” on 03/09/30, Reimer Behrends behrends@cse.msu.edu writes:

This surprises me, since there is no PTR record for 205.178.141.161,
and ruby (1.8.0 in my case) insists on doing a gethostbyaddr() on
that address (in sock_hostbyname()) and will raise a “host not
found” error if gethostbyaddr() fails (usually if there is no PTR
record). That lookup is performed regardless of the value of
Socket.do_not_reverse_lookup.

Incidentally, what is the rationale behind requiring the presence of
a PTR record? There are many hosts that do not have one, especially
virtual web hosts, and I have found Socket/TCPSocket.gethostbyname to be
fairly useless as a result (and instead resorted to using
Resolv.getaddress[es]).

Ah, thanks. Your message reminds me Ruby’s gethostbyname method is
implemented by the combination of getaddrinfo() and gethostbyaddr().
The reason behind this is plain gethostbyname() does not work well
with IPv6 addresses (on some platforms). I want to fix this, but I
don’t know how (yet).

						matz.
11

Does anyone know how to get this to work?

Ask them to correctly register the address, it’s a .COM no ?

Unfortunately any conceivable host could be thrown at the system I’m working
on, so that’s not really an option.

svg% host noblepack.com
noblepack.com has address 205.178.141.161
svg%

svg% host 205.178.141.161
Host 161.141.178.205.in-addr.arpa not found: 3(NXDOMAIN)
svg%

So how come my browser can find it? Can I duplicate in Ruby whatever it is
the browser’s doing to get to find the host?

Thanks,

Nathaniel

<:((><

···

ts [mailto:decoux@moulon.inra.fr] wrote:

12

Hmm, might be Win32 specific problem. Does stopping reverse lookup
make any difference?

Socket.do_not_reverse_lookup = false

[peterv@merlin] ruby -v
ruby 1.8.0 (2003-08-04) [i686-linux]
[peterv@merlin] ruby -rsocket -e ‘Socket.do_not_reverse_lookup=false;p TCPSocket.gethostbyname(“noblepack.com”)’
-e:1:in gethostbyname': host not found (SocketError) from -e:1 [peterv@merlin] ruby -rsocket -e 'Socket.do_not_reverse_lookup=true;p TCPSocket.gethostbyname("noblepack.com")' -e:1:in gethostbyname’: host not found (SocketError)
from -e:1
[peterv@merlin] ping -c 1 noblepack.com
PING noblepack.com (205.178.141.161): 56 data bytes
64 bytes from 205.178.141.161: icmp_seq=0 ttl=110 time=82.2 ms

— noblepack.com ping statistics —
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 82.2/82.2/82.2 ms

So it’s definitely not Win32 specific. Also I would have stopped reverse
lookup by setting Socket.do_not_reverse_lookup = true, but either way it
doesn’t help…

Peter

13

Yukihiro Matsumoto wrote:

Hmm, might be Win32 specific problem. Does stopping reverse lookup
make any difference?
Socket.do_not_reverse_lookup = false

No. Same results.

James

14

  Socket.do_not_reverse_lookup = false

If I'm right ruby don't use it in this case, perhaps a good idea to add
this test.

svg% host noblepack.com
noblepack.com has address 205.178.141.161
svg%

svg% host 205.178.141.161
Host 161.141.178.205.in-addr.arpa not found: 3(NXDOMAIN)
svg%

Guy Decoux

15

Unfortunately any conceivable host could be thrown at the system I'm working
on, so that's not really an option.

Your original question

I'm trying to use TCPSocket.gethostbyname to verify that a given domain
actually exists in DNS.

you can't, just try

moulon% host xxxxyyyyttttuuu.com
xxxxyyyyttttuuu.com has address 64.94.110.11
moulon%

moulon% host aaazertyuiop.com
aaazertyuiop.com has address 64.94.110.11
moulon%

Guy Decoux

16

Hi,

···

In message “Re: TCPSocket.gethostbyname difficulties” on 03/09/29, Peter Peter.Vanbroekhoven@cs.kuleuven.ac.be writes:

[peterv@merlin] ruby -rsocket -e ‘Socket.do_not_reverse_lookup=false;p TCPSocket.gethostbyname(“noblepack.com”)’
-e:1:in `gethostbyname’: host not found (SocketError)
from -e:1

So it’s definitely not Win32 specific. Also I would have stopped reverse
lookup by setting Socket.do_not_reverse_lookup = true, but either way it
doesn’t help…

Thank you for the important information. So there’s something
different from your environment from mine, that makes lookup fail.
Could you tell me how your ruby (and socket extension) compiled?
By yourself, or installed from package?

						matz.
17

Right, this is due to Verisign’s irritating site-finder thingy, but I can
filter those results out if I want to. What about the following, though:

ntalbott@proxytest:~$ host cancer.org
cancer.org has address 209.135.47.118
cancer.org has address 0.0.0.0
ntalbott@proxytest:~$ ruby -rsocket -e ‘p
TCPSocket::gethostbyname(“cancer.org”)’
-e:1:in `gethostbyname’: host not found (SocketError)
from -e:1

Can I make that work?

Nathaniel

<:((><

···

ts [mailto:decoux@moulon.inra.fr] wrote:

I’m trying to use TCPSocket.gethostbyname to verify that a given
domain actually exists in DNS.

you can’t, just try

moulon% host xxxxyyyyttttuuu.com
xxxxyyyyttttuuu.com has address 64.94.110.11
moulon%

moulon% host aaazertyuiop.com
aaazertyuiop.com has address 64.94.110.11
moulon%

18

Thank you for the important information. So there’s something
different from your environment from mine, that makes lookup fail.
Could you tell me how your ruby (and socket extension) compiled?
By yourself, or installed from package?

Compiled from sources using gcc 2.95. But isn’t your environment different
from mine anyhow because you use a different (set of) DNS server?

Could you maybe see what ‘host’ tells you?

[peterv@merlin] host noblepack.com
noblepack.com A 205.178.141.161
[peterv@merlin] host 205.178.141.161
205.178.141.161 does not exist, try again

If it tells you differently (and can do the reverse lookup), it’s almost
certainly a difference in environment, but not how ruby was compiled etc.

BTW, a look at the source seems to tell me that
a) socket.c does not call gethostbyname in the C libs, but calls
getnameinfo and gethostbyaddr instead
b) it tests the do_not_reverse_lookup flag, but not for deciding on doing
the gethostbyaddr (it’s always done). That call however does what
people in this thread called reverse lookup if I understand correctly

Adding some more info to the error message and recompiling socket.c and
rerunning the example tells me the host is not found by the call to
gethostbyaddr. Since the execution got to that point, I guess the forward
lookup works fine. gethostbyaddr is also always called.

I think your DNS server manages to do the reverse lookup while ours can’t.
Is that possible? I don’t know anything about how reverse lookups work…

Peter

19

  ntalbott@proxytest:~$ host cancer.org
  cancer.org has address 209.135.47.118
  cancer.org has address 0.0.0.0

Why do you want to work with something which is broken ?

svg% host 209.135.47.118
Host 118.47.135.209.in-addr.arpa not found: 3(NXDOMAIN)
svg%

Guy Decoux

20

I think your DNS server manages to do the reverse lookup while ours can’t.
Is that possible? I don’t know anything about how reverse lookups work…

I’ve just taken a look at the rfc1034. The section on inverse queries goes
like this:

3.7.2. Inverse queries (Optional)

Name servers may also support inverse queries that map a particular
resource to a domain name or domain names that have that resource. For
example, while a standard query might map a domain name to a SOA RR, the
corresponding inverse query might map the SOA RR back to the domain
name.

Implementation of this service is optional in a name server, but all
name servers must at least be able to understand an inverse query
message and return a not-implemented error response.

The domain system cannot guarantee the completeness or uniqueness of
inverse queries because the domain system is organized by domain name
rather than by host address or any other resource type. Inverse queries
are primarily useful for debugging and database maintenance activities.

Inverse queries may not return the proper TTL, and do not indicate cases
where the identified RR is one of a set (for example, one address for a
host having multiple addresses). Therefore, the RRs returned in inverse
queries should never be cached.

Inverse queries are NOT an acceptable method for mapping host addresses
to host names; use the IN-ADDR.ARPA domain instead.

Basically, inverse lookups are not expected to work, but DNS servers can
provide them for the domain names they know of themselves. While it’s easy
to delegate queries to DNS servers based on the hierarchical naming, it’s
not based on IP address because IP addresses lack the necessary structure.
Anyway, my DNS server doesn’t know noblepack.com directly, so it can’t
tell me that 205.178.141.161 is really noblepack.com, and finding a DNS
server that does is looking for a needle in a haystack. Matz’s DNS server
happened to know about noblepack.com, either directly or cached, and as
such it could say 205.178.141.161 is noblepack.com. Also this changes
since before the weekend my DNS server could tell me 216.239.37.99 is
really google, while now it can’t. Beats me how google got uncached…

Since ruby’s socket lib always calls C function gethostbyaddr in the call
to TCPSocket#gethostbyname, it is bound to fail except for local and
popular names. I don’t know why TCPSocket#gethostbyname doesn’t simply
call C function gethostbyname, but there is probably a good reason for
that.

Peter