Non-blocking io

rubyists-

i’ve got some code for which i need to implement non-blocking io. i do this
so seldomly that i always have to google clr and read about doing it.
considering that ruby’s thread model at times requires nbio it seems like a
‘standard’ method/lib of accomplishing this safely (handling appropriate
signals, etc) should be out there. anyone know of such a beast? right now
i’m looking at

http://raa.ruby-lang.org/list.rhtml?name=io-wait

but would like to avoid c extensions for this particular application.

any reason why something like io-wait would not be included in the standard
dist? portability i’m guessing…

any complete impl of non-blocking io in sources from the RAA that could be
reccomended for study?

thanks.

-a

···

ATTN: please update your address books with address below!

===============================================================================

EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
PHONE :: 303.497.6469
ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
STP :: Solar-Terrestrial Physics Data | NCEI
NGDC :: http://www.ngdc.noaa.gov/
NESDIS :: http://www.nesdis.noaa.gov/
NOAA :: http://www.noaa.gov/
US DOC :: http://www.commerce.gov/

The difference between art and science is that science is what we
understand well enough to explain to a computer.
Art is everything else.
– Donald Knuth, “Discover”

/bin/sh -c ‘for l in ruby perl;do $l -e “print "\x3a\x2d\x29\x0a"”;done’
===============================================================================

any reason why something like io-wait would not be included in the standard
dist? portability i'm guessing...

svg% ruby -vrio/wait -e 'p IO.instance_method("wait")'
ruby 1.8.0 (2003-08-04) [i686-linux]
#<UnboundMethod: IO#wait>
svg%

Guy Decoux

I don’t know if this is a new bug, known bug, or some sort of feature…
When I try to open a fifo, the entire ruby interpreter locks until
something else opens the other “end” of the fifo. Here’s some example
code:

FIFO_FILE=’/tmp/rubyfifo’
mkfifo "#{FIFO_FILE}" unless test(?e, FIFO_FILE)

thread = Thread.new {
puts "Opening fifo for writing"
File.open(FIFO_FILE, ‘w’) { |file|
puts "Starting write loop"
while true
puts "Thread is running"
sleep 1
file.puts "Thread is running"
end
}
}

puts "Opening fifo for reading"
File.open(FIFO_FILE, ‘r’) { |file|
puts "Reading from fifo"
file.each_line { |line|

  puts "Read from fifo: #{line}"

}
}

The only output I see from this is “Opening fifo for writing”.
If the thread is changed to:

thread = Thread.new {
puts "Starting loop"
while true
puts "Thread is running"
sleep 1
end
}

Then instead, I see this output:
Starting loop
Thread is running
Opening fifo for reading

And I don’t see “Thread is running” until I execute this in another
shell:

cat > /tmp/rubyfifo

As soon as I close the cat, the ruby program exits nicely.

Derek Lewis

···

===================================================================
Java Web-Application Developer

  Email    : email@lewisd.com
  Cellular : 604.312.2846
  Website  : http://www.lewisd.com

“If you’ve got a 5000-line JSP page that has “all in one” support
for three input forms and four follow-up screens, all controlled
by “if” statements in scriptlets, well … please don’t show it
to me :-). Its almost dinner time, and I don’t want to lose my
appetite :-).”
- Craig R. McClanahan

great! i had tried -rio-wait. duh!

i’m still having a delima with non-blocking io:

if you use select or ready? to know that you have io, you then have some
options:

  read using gets (line buffered) but you then don't know how may lines to
  read before you will end up blocking the process

  read using read(nil) - this will block the process forever when you are
  reading from a pipe (i am)

  read using read(buf_size) and non-blocking io, but NOT retrying on
  EAGAIN. this seems like the only viable option but it can cause
  truncation if EAGAIN is thrown while a pipe is thinking about sending
  more output but hasn't yet...

what i really need is

buf = fd.read_all_available_bytes_but_do_not_block

i don’t know if one can accomplish this using ready? and non-blocking reads or
not since it is unknowable what ‘all available’ means…

hmmm…

-a

···

On Sat, 13 Dec 2003, ts wrote:

Date: Sat, 13 Dec 2003 02:14:53 +0900
From: ts decoux@moulon.inra.fr
Newsgroups: comp.lang.ruby
Subject: Re: non-blocking io

any reason why something like io-wait would not be included in the standard
dist? portability i’m guessing…

svg% ruby -vrio/wait -e ‘p IO.instance_method(“wait”)’
ruby 1.8.0 (2003-08-04) [i686-linux]
#<UnboundMethod: IO#wait>
svg%

ATTN: please update your address books with address below!

===============================================================================

EMAIL :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
PHONE :: 303.497.6469
ADDRESS :: E/GC2 325 Broadway, Boulder, CO 80305-3328
STP :: Solar-Terrestrial Physics Data | NCEI
NGDC :: http://www.ngdc.noaa.gov/
NESDIS :: http://www.nesdis.noaa.gov/
NOAA :: http://www.noaa.gov/
US DOC :: http://www.commerce.gov/

The difference between art and science is that science is what we
understand well enough to explain to a computer.
Art is everything else.
– Donald Knuth, “Discover”

/bin/sh -c ‘for l in ruby perl;do $l -e “print "\x3a\x2d\x29\x0a"”;done’
===============================================================================

I don't know if this is a new bug, known bug, or some sort of feature...

Something which is intentional can't be called a bug :slight_smile:

When I try to open a fifo, the entire ruby interpreter locks until
something else opens the other "end" of the fifo. Here's some example
code:

You can use IO::NONBLOCK with IO::WRONLY or IO::RDONLY

Guy Decoux

I don’t know if this is a new bug, known bug, or some sort of feature…

Something which is intentional can’t be called a bug :slight_smile:

When I try to open a fifo, the entire ruby interpreter locks until
something else opens the other “end” of the fifo. Here’s some example
code:

You can use IO::NONBLOCK with IO::WRONLY or IO::RDONLY

Should that be File::NONBLOCK and File::RDONLY? I just changed my code to
use those, but it’s still behaving strangly.

Some more sample code based on the previous:

FIFO_FILE=‘/tmp/rubyfifo’
mkfifo "#{FIFO_FILE}" unless test(?e, FIFO_FILE)

thread = Thread.new {
puts “Opening fifo for writing”
File.open(FIFO_FILE, File::NONBLOCK | File::WRONLY) { |file|
puts “Starting write loop”
while true
puts “Thread is running”
sleep 1
file.puts “Thread is running”
end
}
}

puts “Opening fifo for reading”
File.open(FIFO_FILE, File::NONBLOCK | File::RDONLY) { |file|
puts “Reading from fifo”
sleep 2
while !file.eof
line = file.gets(“\n”)
puts “Read from fifo: #{line}”
end
puts “exited read loop”
}

I’m still never getting the “Starting write loop” output, but I do get the
“Reading from fifo”:

lewisd@derlewi:/tmp$ ruby fifo.rb
Opening fifo for writing
Opening fifo for reading
Reading from fifo
exited read loop
lewisd@derlewi:/tmp$

Also, if I comment out the entire thread bit, I get this output:
lewisd@derlewi:/tmp$ ruby fifo.rb
Opening fifo for reading
Reading from fifo
exited read loop
lewisd@derlewi:/tmp$

With a 2 second delay between “Reading…” and “exited…”, as expected.

If, before I start the ruby program, I run this:
$ echo “lala” > /tmp/rubyfifo
The echo blocks, as the fifo hasn’t been opened for reading yet, and when
I run the ruby program, I get this:
lewisd@derlewi:/tmp$ ruby fifo.rb
Opening fifo for reading
Reading from fifo
Read from fifo: lala
exited read loop
lewisd@derlewi:/tmp$

If I remove the “File::NONBLOCK” from the open command, it waits until I
open the fifo, as expected.

I’m not sure how to read from a fifo reliably, without leaving out the
NONBLOCK, which will then block the rest of the program. :frowning:

Guy Decoux

Derek Lewis

···

On Sat, 13 Dec 2003, ts wrote:

===================================================================
Java Web-Application Developer

  Email    : email@lewisd.com
  Cellular : 604.312.2846
  Website  : http://www.lewisd.com

“If you’ve got a 5000-line JSP page that has “all in one” support
for three input forms and four follow-up screens, all controlled
by “if” statements in scriptlets, well … please don’t show it
to me :-). Its almost dinner time, and I don’t want to lose my
appetite :-).”
- Craig R. McClanahan

Should that be File::NONBLOCK and File::RDONLY? I just changed my code to

IO:: is just fine

use those, but it's still behaving strangly.

Some more sample code based on the previous:

FIFO_FILE='/tmp/rubyfifo'
`mkfifo "#{FIFO_FILE}"` unless test(?e, FIFO_FILE)

add this

   Thread.abort_on_exception = true

and you'll see the error

thread = Thread.new {
   puts "Opening fifo for writing"
   File.open(FIFO_FILE, File::NONBLOCK | File::WRONLY) { |file|
      puts "Starting write loop"
      while true
         puts "Thread is running"
         sleep 1
         file.puts "Thread is running"
      end
   }
}

svg% cat b.rb
#!/usr/bin/ruby
FIFO_FILE='/tmp/rubyfifo'
`mkfifo "#{FIFO_FILE}"` unless test(?e, FIFO_FILE)

Thread.abort_on_exception = true
thread = Thread.new {
   begin
      puts "Opening fifo for writing"
      File.open(FIFO_FILE, IO::NONBLOCK | IO::WRONLY) { |file|
         puts "Starting write loop"
         while true
            puts "Thread is running"
            sleep 1
            file.puts "Thread is running"
            file.flush
         end
      }
   rescue
      p $!
      sleep 1
      retry
   end
}

puts "Opening fifo for reading"
File.open(FIFO_FILE, IO::NONBLOCK | IO::RDONLY) { |file|
   puts "Reading from fifo"
   sleep 2
   while !file.eof
      line = file.gets("\n")
      puts "Read from fifo: #{line}"
   end
   puts "exited read loop"
}
svg%

svg% b.rb
Opening fifo for writing
#<Errno::ENXIO: No such device or address - /tmp/rubyfifo>
Opening fifo for reading
Reading from fifo
Opening fifo for writing
Starting write loop
Thread is running
Thread is running
Read from fifo: Thread is running
Thread is running
Read from fifo: Thread is running
Thread is running
Read from fifo: Thread is running
Thread is running
Read from fifo: Thread is running
^Z
[4]+ Stopped b.rb
svg%

Guy Decoux

I tried IO::NONBLOCK:
lewisd@derlewi:/tmp$ ruby
p IO::NONBLOCK
-:1: uninitialized constant NONBLOCK at IO (NameError)

I’m using 1.6.8, is it perhaps only in 1.8?

I feel rather silly about missing that abort_on_exception bit, seeing as I
just recommended the same thing to someone else on the mailing list here.
:wink:

I think understand why I’m getting a “No such device or address” error
opening it for writing, but not while opening it for reading.

It seems like it has to be opened for reading before it can be opened for
writing. Does that sound right? If I put a sleep before the WRONLY open,
so that the RDONLY open happens first, everything seems to work, with the
caveat being that I have to put a file.flush after the file.puts, or I
never seem to read anything in the read loop.

Is there a reason ruby behaves like this, rather than having the open
calls succeede without blocking (provided the file exists), and having the
blocking happen on the read/write calls? My only past experience with
reading/writing fifos was in Java, and that was how it behaved, if memory
serves me correctly.

···

On Sat, 13 Dec 2003, ts wrote:

Should that be File::NONBLOCK and File::RDONLY? I just changed my code to

IO:: is just fine

use those, but it’s still behaving strangly.

Some more sample code based on the previous:

FIFO_FILE=‘/tmp/rubyfifo’
mkfifo "#{FIFO_FILE}" unless test(?e, FIFO_FILE)

add this

Thread.abort_on_exception = true

and you’ll see the error

thread = Thread.new {
puts “Opening fifo for writing”
File.open(FIFO_FILE, File::NONBLOCK | File::WRONLY) { |file|
puts “Starting write loop”
while true
puts “Thread is running”
sleep 1
file.puts “Thread is running”
end
}
}

svg% cat b.rb
#!/usr/bin/ruby
FIFO_FILE=‘/tmp/rubyfifo’
mkfifo "#{FIFO_FILE}" unless test(?e, FIFO_FILE)

Thread.abort_on_exception = true
thread = Thread.new {
begin
puts “Opening fifo for writing”
File.open(FIFO_FILE, IO::NONBLOCK | IO::WRONLY) { |file|
puts “Starting write loop”
while true
puts “Thread is running”
sleep 1
file.puts “Thread is running”
file.flush
end
}
rescue
p $!
sleep 1
retry
end
}

puts “Opening fifo for reading”
File.open(FIFO_FILE, IO::NONBLOCK | IO::RDONLY) { |file|
puts “Reading from fifo”
sleep 2
while !file.eof
line = file.gets(“\n”)
puts “Read from fifo: #{line}”
end
puts “exited read loop”
}
svg%

svg% b.rb
Opening fifo for writing
#<Errno::ENXIO: No such device or address - /tmp/rubyfifo>
Opening fifo for reading
Reading from fifo
Opening fifo for writing
Starting write loop
Thread is running
Thread is running
Read from fifo: Thread is running
Thread is running
Read from fifo: Thread is running
Thread is running
Read from fifo: Thread is running
Thread is running
Read from fifo: Thread is running
^Z
[4]+ Stopped b.rb
svg%

Guy Decoux

Derek Lewis

===================================================================
Java Web-Application Developer

  Email    : email@lewisd.com
  Cellular : 604.312.2846
  Website  : http://www.lewisd.com

“If you’ve got a 5000-line JSP page that has “all in one” support
for three input forms and four follow-up screens, all controlled
by “if” statements in scriptlets, well … please don’t show it
to me :-). Its almost dinner time, and I don’t want to lose my
appetite :-).”
- Craig R. McClanahan

Is there a reason ruby behaves like this, rather than having the open
calls succeede without blocking (provided the file exists), and having the
blocking happen on the read/write calls? My only past experience with
reading/writing fifos was in Java, and that was how it behaved, if memory
serves me correctly.

Well the best question is : Is there a reason linux behave like this
... :slight_smile:

svg% man fifo
FIFO(4) Linux Programmer's Manual FIFO(4)

NAME
       fifo - first-in first-out special file, named pipe
[...]
       A process can open a FIFO in non-blocking mode. In this case, opening
       for read only will succeed even if noone has opened on the write side
       yet; opening for write only will fail with ENXIO (no such device or
       address) unless the other end has already been opened.
[...]
svg%

Guy Decoux

Oh, well I guess that settles it. :slight_smile: Thanks. I should be able to come up
with something workable, now that I know what’s going on.

···

On Sat, 13 Dec 2003, ts wrote:

Is there a reason ruby behaves like this, rather than having the open
calls succeede without blocking (provided the file exists), and having the
blocking happen on the read/write calls? My only past experience with
reading/writing fifos was in Java, and that was how it behaved, if memory
serves me correctly.

Well the best question is : Is there a reason linux behave like this
:slight_smile:

svg% man fifo
FIFO(4) Linux Programmer’s Manual FIFO(4)

NAME
fifo - first-in first-out special file, named pipe
[…]
A process can open a FIFO in non-blocking mode. In this case, opening
for read only will succeed even if noone has opened on the write side
yet; opening for write only will fail with ENXIO (no such device or
address) unless the other end has already been opened.
[…]
svg%

Guy Decoux

Derek Lewis

===================================================================
Java Web-Application Developer

  Email    : email@lewisd.com
  Cellular : 604.312.2846
  Website  : http://www.lewisd.com

“If you’ve got a 5000-line JSP page that has “all in one” support
for three input forms and four follow-up screens, all controlled
by “if” statements in scriptlets, well … please don’t show it
to me :-). Its almost dinner time, and I don’t want to lose my
appetite :-).”
- Craig R. McClanahan