Ruby (1.6.7) Net::FTP/OS call hang

Hello all,

I'm new to this list, and to Ruby as well.  I have read the online 

book “Programming Ruby: The Pragmatic Programmer’s Guide” to the end of
the chapter on threads.

I think Ruby is a great language, and I would love to use it, but for 

one problem. I am creating an FTP program to hone my Ruby skills,
using RubyCocoa as a platform. Everything is working well so far,
except when I connect to an FTP server, all the threads hang. I
remember reading in the book that this is something to be expected in
Ruby, as OS calls like this hang all threads when they dont return
immediately. Obviously this is going to be a huge problem, as all the
benefits of multithreading are moot, when most of my threads will be
doing FTP transactions. My question is: Have and subsequent versions
of Ruby remedied this problem? I understand it is a symptom of how
Ruby works, but knowing that does not help me. I have looked for
changelogs and such, with no luck so far.

Thanks,
Sean

Sean Gilbertson wrote:

I think Ruby is a great language, and I would love to use it, but 

for one problem. I am creating an FTP program to hone my Ruby skills,
using RubyCocoa as a platform. Everything is working well so far,
except when I connect to an FTP server, all the threads hang. I
remember reading in the book that this is something to be expected in
Ruby, as OS calls like this hang all threads when they dont return
immediately. Obviously this is going to be a huge problem, as all the
benefits of multithreading are moot, when most of my threads will be
doing FTP transactions. My question is: Have and subsequent versions of
Ruby remedied this problem? I understand it is a symptom of how Ruby
works, but knowing that does not help me. I have looked for changelogs
and such, with no luck so far.

You’re probably hanging doing the DNS lookup. Perhaps using the thread
aware resolv.rb library might help.

Cheers

Dave

My text book says gethostbyname is bad and you should use
gethostbyname_r to live happy with threads. But I don’t know how to do
this with ruby :confused:

PS
and I don0t even knoiw if this sysacall is available on OSX…

···

il Sat, 3 May 2003 03:51:43 +0900, Sean Gilbertson prell@cfl.rr.com ha scritto::

“Sean Gilbertson” prell@cfl.rr.com wrote in message:

I tested out some file downloads to make sure that those would not
freeze anything, and they didnt seem to at all. Do you know of any
other problem areas that I should look out for?

I don’t know if this will apply to you, but on Win 2K, I have this program
using net/ftp:

···

#------------------------------------------------------------

require "net/ftp"

ftp = Net::FTP::new("ftp-srvr")

ftp.login("anonymous","")
a = ftp.dir("*.*")
puts a if $DEBUG

tb = Time::now
ftp.getbinaryfile("Sales.ppt","junk.bin")
ta = Time::now

puts "Ftp get took #{ta-tb} seconds"

tb = Time::now
ftp.putbinaryfile("junk.bin","junk.bin")
ta = Time::now

puts "Ftp put took #{ta-tb} seconds"

puts ftp.status if $DEBUG
ftp.close

#-----------------------------------------------------

Ran it:

C:\atest>ruby -wv tst_ftp1.rb
ruby 1.6.8 (2002-12-24) [i586-mswin32]
Ftp get took 20.248 seconds
Ftp put took 20.809 seconds

Ran it with profiler:

C:\atest>ruby -wv -rprofile tst_ftp1.rb
ruby 1.6.8 (2002-12-24) [i586-mswin32]
Ftp get took 23.703 seconds
Ftp put took 23.132 seconds
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 85.22    16.51     16.51      808    20.44    20.44  IO#read
  6.97    17.87      1.35        3   450.33  6248.67  Kernel.loop
  2.74    18.40      0.53      814     0.65     0.85  Proc#call
  1.29    18.65      0.25        3    83.33   123.33  Kernel.require
  1.19    18.88      0.23      821     0.28     0.28  IO#write
 [snip]

Notice how IO#read took 85% of the time

Then ran the native ftp client:

C:\atest>ftp ftp-srvr
Connected to ftp-srvr.abc.xyz.com.
220 DBM-Team FTP Server
User (ftp-srvr.abc.xyz.com:(none)): anonymous
331 User name okay, please send complete E-mail address as password.
Password:
230 User logged in, proceed.

binary
200 Type set to I.
get Sales.ppt
200 PORT Command successful.
150 Opening BINARY mode data connection for Sales.ppt (1649152 bytes).
226 Transfer complete.
ftp: 1649152 bytes received in 0.26Seconds 6318.59Kbytes/sec.
put junk.bin
200 PORT Command successful.
150 Opening BINARY mode data connection for junk.bin.
226 Transfer complete.
ftp: 1649152 bytes sent in 0.22Seconds 7462.23Kbytes/sec.
quit
221 Goodbye!

C:\atest>

This was almost 100 times faster !

So I tried with Ruby 1.8 :

C:\atest>ruby -wv tst_ftp1.rb
ruby 1.8.0 (2003-03-03) [i386-mswin32]
Ftp get took 0.511 seconds
Ftp put took 0.831 seconds

Great !
But it is still twice as slow …and with the profiler:

C:\atest>ruby -wv -rprofile tst_ftp1.rb
ruby 1.8.0 (2003-03-03) [i386-mswin32]
Ftp get took 1.533 seconds
Ftp put took 1.722 seconds
  %   cumulative   self              self     total
 time   seconds   seconds    calls  ms/call  ms/call  name
 52.79     1.33      1.33        3   444.00   644.33  Kernel.loop
  9.91     1.58      0.25      808     0.31     0.31  IO#read
  8.76     1.80      0.22      821     0.27     0.27  IO#write
  5.15     1.93      0.13      842     0.15     0.15  String#==
  3.17     2.01      0.08        3    26.67    66.67  Kernel.require
  2.38     2.07      0.06       16     3.75     3.75

MonitorMixin.mon_exit
1.98 2.12 0.05 1 50.00 50.00
Profiler__.start_profile
1.98 2.17 0.05 10 5.00 12.00 Net::FTP#voidresp
1.59 2.21 0.04 15 2.67 2.67 IO#readline
1.59 2.25 0.04 16 2.50 3.13
MonitorMixin.mon_enter
1.59 2.29 0.04 16 2.50 164.56
MonitorMixin.synchronize
1.19 2.32 0.03 15 2.00 4.67 Net::FTP#getline
1.19 2.35 0.03 1 30.00 30.00
TCPSocket#initialize
1.19 2.38 0.03 6 5.00 5.00
Module#attr_accessor
0.79 2.40 0.02 15 1.33 8.00 Net::FTP#getresp
0.79 2.42 0.02 15 1.33 6.00
Net::FTP#getmultiline
0.40 2.43 0.01 1 10.00 40.00 Net::FTP#login
[snip]

And this is with a small 1.6MB file. As the file size grows (I tried with
400 MB file)
the difference is staggering.

Something to think about, eh ?
– shanko

I am now happily using Resolv.rb with successful results. Thank you
very much!

I tested out some file downloads to make sure that those would not 

freeze anything, and they didnt seem to at all. Do you know of any
other problem areas that I should look out for?

Thanks again,
Sean

···

On Friday, May 2, 2003, at 03:54 PM, Dave Thomas wrote:

Sean Gilbertson wrote:

I think Ruby is a great language, and I would love to use it, but 

for one problem. I am creating an FTP program to hone my Ruby
skills, using RubyCocoa as a platform. Everything is working well so
far, except when I connect to an FTP server, all the threads hang. I
remember reading in the book that this is something to be expected in
Ruby, as OS calls like this hang all threads when they dont return
immediately. Obviously this is going to be a huge problem, as all
the benefits of multithreading are moot, when most of my threads will
be doing FTP transactions. My question is: Have and subsequent
versions of Ruby remedied this problem? I understand it is a symptom
of how Ruby works, but knowing that does not help me. I have looked
for changelogs and such, with no luck so far.

You’re probably hanging doing the DNS lookup. Perhaps using the thread
aware resolv.rb library might help.

Cheers

Dave

Your textbook is talking about OS (native) threads - which Ruby doesn’t use
anyway.

(The _r functions don’t use statically-allocated buffers, which could be
overwritten if two threads made the same call at the same time; instead they
write their results into space allocated by the caller)

Regards,

Brian.

···

On Sat, May 03, 2003 at 05:28:48AM +0900, gabriele renzi wrote:

il Sat, 3 May 2003 03:51:43 +0900, Sean Gilbertson prell@cfl.rr.com > ha scritto::

My text book says gethostbyname is bad and you should use
gethostbyname_r to live happy with threads.

Sorry everyone; I didn’t mean to send this to the whole list. I’ll
try not to let it happen again.

···

On Friday, May 2, 2003, at 05:20 PM, Sean Gilbertson wrote:

I am now happily using Resolv.rb with successful results. Thank you
very much!

I tested out some file downloads to make sure that those would not
freeze anything, and they didnt seem to at all. Do you know of any
other problem areas that I should look out for?

Thanks again,
Sean

On Friday, May 2, 2003, at 03:54 PM, Dave Thomas wrote:

Sean Gilbertson wrote:

I think Ruby is a great language, and I would love to use it, 

but for one problem. I am creating an FTP program to hone my Ruby
skills, using RubyCocoa as a platform. Everything is working well
so far, except when I connect to an FTP server, all the threads
hang. I remember reading in the book that this is something to be
expected in Ruby, as OS calls like this hang all threads when they
dont return immediately. Obviously this is going to be a huge
problem, as all the benefits of multithreading are moot, when most
of my threads will be doing FTP transactions. My question is: Have
and subsequent versions of Ruby remedied this problem? I understand
it is a symptom of how Ruby works, but knowing that does not help
me. I have looked for changelogs and such, with no luck so far.

You’re probably hanging doing the DNS lookup. Perhaps using the
thread aware resolv.rb library might help.

Cheers

Dave

Though it has been Dave who suggested to use resolv.rb, there is a
(admittedly very small :slight_smile: possibility that this wouldn’t have solved
your problem. So at least I am interested in feedback like this.

Regards,
Pit

···

On 3 May 2003 at 6:29, Sean Gilbertson wrote:

Sorry everyone; I didn’t mean to send this to the whole list. I’ll
try not to let it happen again.

On Friday, May 2, 2003, at 05:20 PM, Sean Gilbertson wrote:

I am now happily using Resolv.rb with successful results.  Thank you 

very much!

I tested out some file downloads to make sure that those would not 

freeze anything, and they didnt seem to at all. Do you know of any
other problem areas that I should look out for?