How can I perform a nonblocking IO read? That is, read whatever is
there to read and return. If there's nothing yet, just return.
Failing that, is there a way to manually check if there's anything to
read or the read will block?
(Note that I'm developing for both 'nix and Windows.)
it can't be done. search the archives, this sort of thing almost always
indicates a design flaw. for instance - what will your program do if there is
no input?
regards.
-a
On Wed, 1 Nov 2006 srobertjames@gmail.com wrote:
How can I perform a nonblocking IO read? That is, read whatever is
there to read and return. If there's nothing yet, just return.Failing that, is there a way to manually check if there's anything to
read or the read will block?(Note that I'm developing for both 'nix and Windows.)
--
my religion is very simple. my religion is kindness. -- the dalai lama
Are you having an interactive conversation with the process created by
popen4? Is that why you need nbio, because you need to poll more than one
stream simultaneously?
You can get the file descriptors associated with stdout and stderr (they'll
be 1 and 2 unless you did something unusual), set them nonblocking, and
select them for reading with a short timeout.
On 10/31/06, srobertjames@gmail.com <srobertjames@gmail.com> wrote:
How can I perform a nonblocking IO read? That is, read whatever is
there to read and return. If there's nothing yet, just return.Failing that, is there a way to manually check if there's anything to
read or the read will block?(Note that I'm developing for both 'nix and Windows.)
How can I perform a nonblocking IO read? That is, read whatever is there to read and return. If there's nothing yet, just return.
That's what the Unix read() function does ("man 2 read"), available in Ruby as IO#sysread(). The ri documentation doesn't explain much about how to use Ruby's sysread, unfortunately.
Failing that, is there a way to manually check if there's anything to read or the read will block?
(Note that I'm developing for both 'nix and Windows.)
The select() function (Kernel#select, in Ruby) is the standard way to do IO multiplexing under Unix. You give it a list of file handles on which you're waiting to be able to read or write, and it returns when one or more of them is ready. Windows provides a select() function, too, but it only supports sockets (not pipes or ordinary file IO.) The Win32 API generally doesn't provide non-blocking IO methods, because it's assumed you'll use threads if you want to do something else while waiting for IO.
it can't be done. search the archives, this sort of thing almost always
indicates a design flaw. for instance - what will your program do if there is
no input?
Check other IO channels, wait for other sorts of events, do work - whatever. Where do come from asserting that the use of non-blocking IO represents a "design flaw"?
Tom
On Oct 31, 2006, at 6:23 PM, ara.t.howard@noaa.gov wrote:
On Wed, 1 Nov 2006 srobertjames@gmail.com wrote:
I'm running an external command (using popen4 - yes, it is available on
Windows also). I capture stdout and stderr. "what will your program do
if there is no input?" - I assume you mean output - Well, I hope there
is no stderr output. If there is, I'd like to capture it and feed it
to Logger.
Likewise, if there is a problem, and there's no stdout, then I don't
want to wait for ever - I want to throw an exception.
ara.t.howard@noaa.gov wrote:
On Wed, 1 Nov 2006 srobertjames@gmail.com wrote:
> How can I perform a nonblocking IO read? That is, read whatever is
> there to read and return. If there's nothing yet, just return.
>
> Failing that, is there a way to manually check if there's anything to
> read or the read will block?
>
> (Note that I'm developing for both 'nix and Windows.)it can't be done. search the archives, this sort of thing almost always
indicates a design flaw. for instance - what will your program do if there is
no input?
[ara.t.howard@noaa.gov]
it can't be done. search the archives, this sort of thing almost
always indicates a design flaw. for instance - what will your
program do if there is no input?
nonblocking IO is not a design-flaw; it is something that is in common
use and is supported by most languages with a decent socket API. on
some OS'es this is even available for file IO (although widespread
language support seems to lag behind).
Java for example got proper nonblocking socket IO in 1.4 (see
JSR-000051 "New I/O APIs for the JavaTM Platform").
for examples of use you might want to check out the Reactor pattern
and other patterns for concurrent programming.
-Bjørn
I have a messaging middleware server running right now that is
processing millions of messages a day. It's written in Ruby, and does
_all_ it's work in a single thread using IO multiplexing with select().
Not only can it be done - it is fairly easy (~700 lines for the entire
app, including db persistence support etc.). Doing the same thing with
threads and blocking IO, on the other hand, would fall apart horribly
due to the way Ruby does threading. Forking also wouldn't be an option
as the processes would still need to actually exchange those messages.
In fact, internally, Ruby does all it's IO using non-blocking IO
exactly because the threading model would cause everything to block
otherwise. Incidentally that's also one of the reasons why using
threads + blocking IO performs extremely badly in Ruby once the number
of threads gets beyond a certain level, because it causes Ruby to call
select() far too often.
Vidar
ara.t.howard@noaa.gov wrote:
On Wed, 1 Nov 2006 srobertjames@gmail.com wrote:
> How can I perform a nonblocking IO read? That is, read whatever is
> there to read and return. If there's nothing yet, just return.
>
> Failing that, is there a way to manually check if there's anything to
> read or the read will block?
>
> (Note that I'm developing for both 'nix and Windows.)it can't be done. search the archives, this sort of thing almost always
indicates a design flaw. for instance - what will your program do if there is
no input?
In the interests of promoting creativity, I'll ask the question without
any prior assumptions:
I'm using popen4 (great gem, btw) to run an external command and
capture stdout and stderr. If one (or both) of those is empty, I don't
want to hang. Just keep on moving. In fact, stderr will normally be
empty.
How could I do this?
(I'm supporting 'nix and Windows)
How can I perform a nonblocking IO read? That is, read whatever is there to read and return. If there's nothing yet, just return.
That's what the Unix read() function does ("man 2 read"), available in Ruby as IO#sysread(). The ri documentation doesn't explain much about how to use Ruby's sysread, unfortunately.
no, that blocks. try this:
harp:~ > ruby -e 'STDIN.sysread(1)'
it blocks. the OP wants a non-blocking call. eg
harp:~ > ruby -r io/nonblock -e 'STDIN.nonblock{ STDIN.sysread(1) }'
-e:1:in `sysread': Resource temporarily unavailable (Errno::EAGAIN)
from -e:1
from /home/ahoward//lib/ruby/1.8/io/nonblock.rb:19:in `nonblock'
from -e:1
The select() function (Kernel#select, in Ruby) is the standard way to do IO multiplexing under Unix. You give it a list of file handles on which you're waiting to be able to read or write, and it returns when one or more of them is ready. Windows provides a select() function, too, but it only supports sockets (not pipes or ordinary file IO.) The Win32 API generally doesn't provide non-blocking IO methods, because it's assumed you'll use threads if you want to do something else while waiting for IO.
right. but ruby's thread are green and, on windows, block the entire process
when waiting on IO so not an option here.
Check other IO channels, wait for other sorts of events, do work - whatever. Where do come from asserting that the use of non-blocking IO represents a "design flaw"?
because nine times out of ten code has nothing to do if no input is available.
if it does that's another story, but then there are basically two options:
- use select or an event abstraction in one giant loop. if you do this you are
going to block anyhow, only based on all inputs, not just one.
- use threads, only you can't if you're wanting ruby code to run nb in windows.
i'm not saying there is never a case for nbio - i use it myself - just that,
unless a developer has a good reason to be doing something else it's not
required and, as a search of ruby-talk will show, that is often the case.
regards.
-a
On Wed, 1 Nov 2006, Tom Pollard wrote:
On Oct 31, 2006, at 6:23 PM, ara.t.howard@noaa.gov wrote:
On Wed, 1 Nov 2006 srobertjames@gmail.com wrote:
--
my religion is very simple. my religion is kindness. -- the dalai lama
so, basically you want to timeout on a read right? in unix you can just use
require 'timeout'
Timeout::timeout(n) do
Open4.popen4 ...
...
end
but, this is implimented via a thread so i think the read is going to block
your entire process in windows... you might try it.
-a
On Wed, 1 Nov 2006 srobertjames@gmail.com wrote:
I'm running an external command (using popen4 - yes, it is available on
Windows also). I capture stdout and stderr. "what will your program do if
there is no input?" - I assume you mean output - Well, I hope there is no
stderr output. If there is, I'd like to capture it and feed it to Logger.Likewise, if there is a problem, and there's no stdout, then I don't want to
wait for ever - I want to throw an exception.
--
my religion is very simple. my religion is kindness. -- the dalai lama
At a very deep level (last time I looked) thread animation and IO met
and became the same thing in a giant select lurking in the core of the
ruby Interpreter.
ie. Use select / spin another thread amounted cosmically to the same
thing. Choose whatever is easiest for you.
John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : john.carter@tait.co.nz
New Zealand
"We have more to fear from
The Bungling of the Incompetent
Than from the Machinations of the Wicked." (source unknown)
On Wed, 1 Nov 2006, Tom Pollard wrote:
The select() function (Kernel#select, in Ruby) is the standard way to do IO multiplexing under Unix.
[Tom Pollard <tomp@earthlink.net>]
The select() function (Kernel#select, in Ruby) is the standard way to
do IO multiplexing under Unix. You give it a list of file handles on
which you're waiting to be able to read or write, and it returns when
one or more of them is ready. Windows provides a select() function,
too, but it only supports sockets (not pipes or ordinary file IO.)
if memory serves, a select(2) on a file descriptors representing a
(disk) file only had defined semantics when used to wait on mandatory
locking and is not used for ensuring that a read(2) or write(2) will
not block. (I might be in error since it has been a while since I
looked into it).
-Bjørn
[ara.t.howard@noaa.gov]
>
> it can't be done. search the archives, this sort of thing almost
> always indicates a design flaw. for instance - what will your
> program do if there is no input?nonblocking IO is not a design-flaw; it is something that is in common
use and is supported by most languages with a decent socket API. on
some OS'es this is even available for file IO (although widespread
language support seems to lag behind).Java for example got proper nonblocking socket IO in 1.4 (see
JSR-000051 "New I/O APIs for the JavaTM Platform").
but it's archaic compared to event driven frameworks like
libevent
http://liboop.ofb.net/
in addition it's a hack in the c lib of many oses.
for examples of use you might want to check out the Reactor pattern
and other patterns for concurrent programming.
afaik the reactor pattern is a synchronous pattern
artima - Comparing Two High-Performance I/O Design Patterns
not unlike the model of libevent and liboop - which are both synchronous...
am i missing something?
-a
On Wed, 1 Nov 2006, Bjorn Borud wrote:
--
my religion is very simple. my religion is kindness. -- the dalai lama
Not very attractive...
Back to the original question: there is no simple way to read
nonblocking, or see if any output is available ?!?!
ara.t.howard@noaa.gov wrote:
On Wed, 1 Nov 2006 srobertjames@gmail.com wrote:
> I'm running an external command (using popen4 - yes, it is available on
> Windows also). I capture stdout and stderr. "what will your program do if
> there is no input?" - I assume you mean output - Well, I hope there is no
> stderr output. If there is, I'd like to capture it and feed it to Logger.
>
> Likewise, if there is a problem, and there's no stdout, then I don't want to
> wait for ever - I want to throw an exception.so, basically you want to timeout on a read right? in unix you can just use
require 'timeout'
Timeout::timeout(n) do
Open4.popen4 ...
...
...
endbut, this is implimented via a thread so i think the read is going to block
your entire process in windows... you might try it.-a
--
my religion is very simple. my religion is kindness. -- the dalai lama
Have you tried simply reading from them? When the command terminates, that should close those pipes on the command's end. At that point, a simple read on the stdout handle, for instance, will return whatever output there was and then eof. If there's no output, the read should return immediately. Did you try that yet? Did you try the (Windows-oriented) example from the 'ri Open4' page? If you're working in Unix you could try
status = POpen4::poen4('cat -') do |stdout, stderr, stdin|
stdin.puts 'hello world'
stdin.close
puts "stdout: #{stdout.read.strip}"
puts "stderr: #{stderr.read.strip}"
end
Anyway, you would only need nonblocking IO if you wanted to read bits of the stderr stream before the command exited, but that doesn't sound like what you're want.
Tom
On Oct 31, 2006, at 7:20 PM, srobertjames@gmail.com wrote:
In the interests of promoting creativity, I'll ask the question without
any prior assumptions:I'm using popen4 (great gem, btw) to run an external command and
capture stdout and stderr. If one (or both) of those is empty, I don't
want to hang. Just keep on moving. In fact, stderr will normally be
empty.How could I do this?
Ara, I'm curious to know why you think the usage of nbio is archaic or
indicative of a design flaw. I wrote the EventMachine library in order to
make certain kinds of Ruby programs easier to write, and it uses nbio
pervasively. I'm fairly sure libevent does, also. EM works with almost any
kind of descriptor, except for certain things on Windows (like _pipe()
descriptors, which are nonselectable, and some of the native Windows objects
that don't support a socket-like API).
Up until recently (late May 2006), Ruby didn't have complete support for
nbio on all descriptor types, which is one of the reasons that EventMachine
includes a compiled C++ extension. EM interoperates perfectly well with Ruby
threads,* which is another important reason to use nbio.
On 10/31/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:
On Wed, 1 Nov 2006, Bjorn Borud wrote:
> [ara.t.howard@noaa.gov]
> >
> > it can't be done. search the archives, this sort of thing almost
> > always indicates a design flaw. for instance - what will your
> > program do if there is no input?
>
> nonblocking IO is not a design-flaw; it is something that is in common
> use and is supported by most languages with a decent socket API. on
> some OS'es this is even available for file IO (although widespread
> language support seems to lag behind).
>
> Java for example got proper nonblocking socket IO in 1.4 (see
> JSR-000051 "New I/O APIs for the JavaTM Platform").but it's archaic compared to event driven frameworks like
---
*Except for certain cases on Windows where Ruby code that accesses standard
descriptors blocks all other Ruby threads. But this is not an EM issue,
In the interests of promoting creativity, I'll ask the question without
any prior assumptions:I'm using popen4 (great gem, btw) to run an external command and
capture stdout and stderr. If one (or both) of those is empty, I don't
want to hang. Just keep on moving. In fact, stderr will normally be
empty.How could I do this?
(I'm supporting 'nix and Windows)
Windows will be a problem. Admittedly, I haven't tried
ruby 1.8.5 yet, which has new nonblock_* methods. However,
my expectation is that you'll only get nonblocking behavior
on windows from sockets, not from pipes.
On Windows, calling select() on a pipe, always returns immediately with "data ready to read", regardless if there's
any data there or not.
This has been the bane of my existence on Windows ruby for
5 or 6 years. I do IPC on Windows ruby using TCP over
loopback, instead of pipes, in order to get nonblocking
semantics. (That still doesn't help for reading from the
console, though... (search the archives for 'kbhit' for a
partial solution there...))
One of these years, I'd like to chat with a Windows guru
and ask how he/she would recommend making a select() that
works on both sockets and pipes on Windows. Ruby could
*really* use one.
Regards,
Bill
From: <srobertjames@gmail.com>
I thought Ruby internally uses non-blocking I/O in order to avoid that a green thread reading something blocks every other thread: am I wrong?
Or is this true just under unix?
Il giorno 01/nov/06, alle ore 01:33, ara.t.howard@noaa.gov ha scritto:
The select() function (Kernel#select, in Ruby) is the standard way to do IO multiplexing under Unix. You give it a list of file handles on which you're waiting to be able to read or write, and it returns when one or more of them is ready. Windows provides a select() function, too, but it only supports sockets (not pipes or ordinary file IO.) The Win32 API generally doesn't provide non-blocking IO methods, because it's assumed you'll use threads if you want to do something else while waiting for IO.
right. but ruby's thread are green and, on windows, block the entire process
when waiting on IO so not an option here.
[ara.t.howard@noaa.gov]
>
> Java for example got proper nonblocking socket IO in 1.4 (see
> JSR-000051 "New I/O APIs for the JavaTM Platform").but it's archaic compared to event driven frameworks like
you are comparing apples and oranges here. you have to look at where
these things fit into the big picture. the above mentioned libraries
provide abstractions that are used to provide a more convenient
programming model, and at least in the case of libevent, provide an
abstraction that insulates code from having to deal with supporting
the various readiness selection APIs that exist. they do not replace
the underlying OS interfaces upon which they are built.
JSR-000051 does more or less the same thing, but with a slightly
different goal in mind. the primary goal is to provide the programmer
with access to asynchronous network IO in a platform-agnostic
manner. the emphasis is on providing a sensible abstraction that works
well with the various operating system APIs that exist -- exposing a
reasonable common subset of functionality that one can expect to be
able to support on a variety of platforms. JSR-000051 provides the
primitives on top of which you would implement IO frameworks that
provide conventient programming models.
for a language, providing sensible APIs for primitives is more
important than imposing a particular programming model. given a set
of sensible primitives that can be widely supported, whatever higher
level frameworks one wishes to create can be built atop that.
in addition it's a hack in the c lib of many oses.
I am not sure I understand exactly what you mean. the way I see it
there are three distinct levels to networking APIs:
1. OS API, that is, the system calls
2. standardized system libraries, like libc, java.io, java.util.nio
etc, which provide access to IO primitives and possibly abstract
away the underlying OS APIs
3. higher level IO frameworks that provide more convenient programming
models
libevent would overlap with both 2 and 3 in this case since its
mission is *both* to abstract away the underlying OS interfaces *and*
provide a convenient programming model. Java's NIO is what you'd find
in 2 and a typical Reactor pattern implementation would be in 3.
(if you use low-level socket APIs (ie the types of functions
documented in section 2 of the UNIX manual pages) on UNIX you will
find yourself using a mix of 1 and 2 if you use C/C++ since you use
wrappers in libc to perform system calls, but this is just a very,
very thin convenience layer on top of the system calls. if this is
confusing then just forget I mentioned it :-).
> for examples of use you might want to check out the Reactor pattern
> and other patterns for concurrent programming.afaik the reactor pattern is a synchronous pattern
no, the Reactor pattern is mainly used to implement asynchronous IO
and is not really anything new. I've both written and seen variations
of this pattern in a multitude of languages since I started writing
networking code in the early 90s and the only thing that has really
changed is that we've gotten better at classifying these types of
patterns, give them better definitions, and give them names. (when I
started writing networking software in the early 90s, "patterns"
wasn't commonly part of the programmers vocabulary).
artima - Comparing Two High-Performance I/O Design Patterns
not unlike the model of libevent and liboop - which are both synchronous...
am i missing something?
you are missing an "a" in front of "synchronous" 
-Bjørn
in io/wait:
--- IO#ready?
returns non-nil if input available without blocking, or nil.
also: ri IO.read_nonblock
On Oct 31, 2006, at 5:30 PM, S. Robert James wrote:
Not very attractive...
Back to the original question: there is no simple way to read
nonblocking, or see if any output is available ?!?!