Select strange behavier

[Simon_Strandgaard]

‘select’ is suppose to watch some file-descriptors and when an event
occurs then an array is returned containing the affected file-descriptors.
Problem: the array I get returned seems contain something different!!!

To me it seems like the array contains non-affected file-descriptors ???

Am I mistaken?.. notify when theres data to read… how should I do?

cat b.rb
require ‘stringio’
require ‘singleton’

class Redir
include Singleton
def initialize
@out = $stdout
@error = $stderr
end
def set_out(elsewhere)
@out = elsewhere
end
def set_error(elsewhere)
@error = elsewhere
end
def write_out(msg)
msg = “” + msg + “”
@out.write(msg)
end
def write_error(msg)
msg = “” + msg + “”
@error.write(msg)
end
end

def system(command)
# todo: stdin
r1, w1 = IO.pipe # stdout
r2, w2 = IO.pipe # stderr
thread = Thread.new do
loop do
puts “waiting”
res = select([r1, r2], nil, nil, 0.1)[0]
p res
if res.include?(r1)
unless r1.eof
Redir.instance.write_out r1.read
else
puts “r1 not”
end
end
if res.include?(r2)
unless r2.eof
Redir.instance.write_error r2.read
else
puts “r2 not”
end
end
end
end
#sleep 0.3 # if we sleep then no output is captured, why ???
pid = fork do
r1.close; r2.close
$defout.reopen(w1); $stdout.reopen(w1); $stderr.reopen(w2)
exec(command)
# todo: should I care closing w1+w2 correctly?
end
w1.close; w2.close
Process.waitpid(pid)
# todo: Wait until the pipes r1+r2 is flushed completly
sleep 0.5
r1.close; r2.close
thread.kill
end

redir = Redir.instance
s = “”
buf = StringIO.new(s, “w”)
redir.set_out(buf)
redir.set_error(buf)
system(“ruby -e ‘puts "hello"; system("ls main.cpp nonexist")’”)

puts “1”
puts s.split(/\n/).collect { |msg| “OK: “+msg }.join(”\n”)
puts “2”

ruby -w b.rb
/home/neoneye/stow/ruby18_r2/lib/ruby/1.8/singleton.rb:145: warning: redefine instance
waiting
[#IO:0x80fcd54]
waiting
[#IO:0x80fcd54, #IO:0x80fcd18]
r1 not
waiting
[#IO:0x80fcd54, #IO:0x80fcd18]
r1 not
r2 not
SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP
SNIP repeats itself many times! SNIP
SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP
waiting
[#IO:0x80fcd54, #IO:0x80fcd18]
r1 not
r2 not
waiting
[#IO:0x80fcd54, #IO:0x80fcd18]
r1 not
r2 not
waiting
[#IO:0x80fcd54, #IO:0x80fcd18]
r1 not
r2 not
waiting
1
OK: hello
OK: main.cpp
OK: ls: nonexist: No such file or directory
OK:
2

ruby/machine

ruby -v
ruby 1.8.0 (2003-03-03) [i386-freebsd5.0]
uname -a
FreeBSD P350.home 5.0-RELEASE FreeBSD 5.0-RELEASE #0: Thu Jan 16 22:16:53 GMT 2003 root@hollin.btc.adaptec.com:/usr/obj/usr/src/sys/GENERIC i386

···

–
Simon Strandgaard

[ts1 (ts)]

                        res = select([r1, r2], nil, nil, 0.1)[0]
                        p res
                        if res.include?(r1)
                                unless r1.eof

svg% man select
[...]
       Three independent sets of descriptors are watched. Those listed in
       readfds will be watched to see if characters become available for read-
       ing (more precisely, to see if a read will not block - in particular, a
       file descriptor is also ready on end-of-file), those in writefds will
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[...]
svg%

Guy Decoux

[Simon_Strandgaard]

My manpages is different… EOF hmmmm.

But then it makes no sense for me to use ‘select’… it will just behave the
same as if i were using polling.

How do I get notify when there is data available (without polling) ?

···

On Sat, 17 May 2003 23:14:47 +0900, ts wrote:

   Three independent sets of descriptors are  watched.   Those  listed  in
   readfds will be watched to see if characters become available for read-
   ing (more precisely, to see if a read will not block - in particular, a
   file  descriptor  is also ready on end-of-file), those in writefds will
   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

–
Simon Strandgaard

BTW: what happened to ‘pigeon’ ?

[Simon_Strandgaard]

Can someone explain why ‘select’ has such behavier and why its smart ?

To me this is non-intuitive… so non-intuitive that I simply don’t
understand it

···

On Sat, 17 May 2003 23:14:47 +0900, ts wrote:

svg% man select
[…]
Three independent sets of descriptors are watched. Those listed in
readfds will be watched to see if characters become available for read-
ing (more precisely, to see if a read will not block - in particular, a
file descriptor is also ready on end-of-file), those in writefds will
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[…]
svg%

–
Simon Strandgaard

BTW: what happened to ‘pigeon’ ?

[ts1 (ts)]

But then it makes no sense for me to use 'select'.. it will just behave the
same as if i were using polling.

IO#read return nil at EOF

Guy Decoux

[Marcin_Qrczak_Kowalc (Marcin 'Qrczak' Kowalczyk)]

Dnia nie 18. maja 2003 11:49, Simon Strandgaard napisa³:

in particular, a file descriptor is also ready on end-of-file),

Can someone explain why ‘select’ has such behavier and why its smart ?

To me this is non-intuitive… so non-intuitive that I simply don’t
understand it

For me it’s very intuitive. How would you otherwise distinguish between end of
stream and a state where it’s not known yet whether there is something more
to read?

select selects descriptors from which you could read without being blocked.
It’s very purpose is to check which of several file descriptors won’t block
(in case of a single descriptor you can just perform a blocking read) or to
block not longer than a specific timeout. In ancient times without select the
only option was busy waiting.

And it’s like C select does.

···

–
__("< Marcin Kowalczyk
__/ qrczak@knm.org.pl
^^ Blog człowieka poczciwego.

[Simon_Strandgaard]

OK.

The code you see in the initial post was/is an attempt to make
the following code: buffered.

thread1 = Thread.new do  # redirect stdout
	loop do
		s = r1.read(1) # TODO: can I make this buffered?
		Redir.instance.write_out(s)
	end
end 

Any suggestions on how buffering can be done ???

···

On Sat, 17 May 2003 23:58:17 +0900, ts wrote:

But then it makes no sense for me to use ‘select’… it will just behave the
same as if i were using polling.

IO#read return nil at EOF

–
Simon Strandgaard

[Simon_Strandgaard]

^^^^^^^^^^^^

Busy waiting is what everyone wants to avoid…
The same in my ‘select’ case, because of EOF notify then the thread
behaves exactly as if I were busy waiting.

I don’t want polling, I want blocking!

Any suggestions to how I should get rid of this busy waiting ???

···

On Sun, 18 May 2003 21:41:36 +0900, Marcin ‘Qrczak’ Kowalczyk wrote:

block not longer than a specific timeout. In ancient times without select the
only option was busy waiting.
–
Simon Strandgaard

[ts1 (ts)]

The code you see in the initial post was/is an attempt to make
the following code: buffered.

The code is fine (you don't need to test for EOF)

IO#read return nil at EOF

IO#write(nil) call nil.to_s which give a String with length == 0
==> IO#write do nothing

Guy Decoux

[Mauricio_Fernndez]

block not longer than a specific timeout. In ancient times without select the
only option was busy waiting.
^^^^^^^^^^^^

Busy waiting is what everyone wants to avoid…
The same in my ‘select’ case, because of EOF notify then the thread
behaves exactly as if I were busy waiting.

I don’t want polling, I want blocking!

Any suggestions to how I should get rid of this busy waiting ???

Essentially removing the IO object that got to EOF from the array of
objects to be examined:

(slight modification of the code you posted)

batsman@tux-chan:/tmp$ expand -t2 aj.rb
def system(command)
r1, w1 = IO.pipe # stdout
r2, w2 = IO.pipe # stderr
fdes = [r1,r2]
thread = Thread.new do
loop do
res = select(fdes, nil, nil, 0.1)[0]
p res
if res.include?(r1)
unless r1.eof
Redir.instance.write_out r1.read
else
fdes.delete r1
thread.kill if fdes ==
end
end
if res.include?(r2)
unless r2.eof
Redir.instance.write_error r2.read
else
fdes.delete r2
thread.kill if fdes ==
end
end
end
end
#sleep 0.3 # if we sleep then no output is captured, why ???
pid = fork do
r1.close; r2.close
$defout.reopen(w1); $stdout.reopen(w1); $stderr.reopen(w2)
exec(command)
# todo: should I care closing w1+w2 correctly?
end
w1.close; w2.close
Process.waitpid(pid)

todo: Wait until the pipes r1+r2 is flushed completly

sleep 0.5
r1.close; r2.close
thread.kill
end

···

On Sun, May 18, 2003 at 09:09:22PM +0900, Simon Strandgaard wrote:

On Sun, 18 May 2003 21:41:36 +0900, Marcin ‘Qrczak’ Kowalczyk wrote:

–
Simon Strandgaard

–
_ _

__ __ | | ___ _ __ ___ __ _ _ __
'_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |__ \ | | | | | (| | | | |
.__/ _,|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

It’s easy to get on the internet and forget you have a life
– Topic on #LinuxGER

[Marcin_Qrczak_Kowalc (Marcin 'Qrczak' Kowalczyk)]

Dnia nie 18. maja 2003 14:09, Simon Strandgaard napisa³:

Busy waiting is what everyone wants to avoid…
The same in my ‘select’ case, because of EOF notify then the thread
behaves exactly as if I were busy waiting.

As I said, don’t try to read any more from a file which said EOF once.
Don’t select on it, forget it.

···

–
__("< Marcin Kowalczyk
__/ qrczak@knm.org.pl
^^ Blog człowieka poczciwego.

[Simon_Strandgaard]

Are you suggestion that I should do like this.
This is polling… its blocking im interested in… how ?

thread = Thread.new do
	loop do
		if (data = r1.read(100))
			Redir.instance.write_out data
		else
			puts "r1 not"
		end
		if (data = r2.read(100))
			Redir.instance.write_error data
		else
			puts "r2 not"
		end
	end
end

···

On Sun, 18 May 2003 00:30:58 +0900, ts wrote:

The code you see in the initial post was/is an attempt to make
the following code: buffered.

IO#read return nil at EOF

–
Simon Strandgaard

[Simon_Strandgaard]

Any suggestions to how I should get rid of this busy waiting ???

I have tried your suggested code and it seems to work, thanks Mauricio

This piece of code is very interesting.

    else
      fdes.delete r1
      thread.kill if fdes == []
    end

if EOF then the filedescriptor is removed from the list. nice!
I had no luck on google… but this answer is great

···

On Mon, 19 May 2003 02:14:21 +0900, Mauricio Fernández wrote:

On Sun, May 18, 2003 at 09:09:22PM +0900, Simon Strandgaard wrote:

–
Simon Strandgaard

[Marcin_Qrczak_Kowalc (Marcin 'Qrczak' Kowalczyk)]

Dnia sob 17. maja 2003 17:05, Simon Strandgaard napisa³:

  loop do
  	if (data = r1.read(100))
  		Redir.instance.write_out data
  	else
  		puts "r1 not"
  	end

If data is nil, then it’s EOF, so you should better not try to read further.
(If the input is a terminal, it will try to input more despite the user
hitting ^D, which is annoying.)

···

–
__("< Marcin Kowalczyk
__/ qrczak@knm.org.pl
^^ Blog człowieka poczciwego.

[Simon_Strandgaard]

replacement for ‘Kernel#system’ which captures all output.
This is the almost final version… this may have others interest
Warning: ‘stdin’ is not yet supported.

Suggestions for improvements is welcome.

def system(command, *args)
#todo: stdin
r1, w1 = IO.pipe # stdout
r2, w2 = IO.pipe # stderr
fdes = [r1, r2]
thread = Thread.new do
loop do
#puts "waiting"
res = select(fdes, nil, nil, nil)[0]
#p res
if res.include?(r1)
#puts "r1"
unless r1.eof
Redir.instance.write_out r1.read
else
#puts "kill r1"
fdes.delete r1
thread.kill if fdes == []
end
end
if res.include?(r2)
#puts "r2"
unless r2.eof
Redir.instance.write_error r2.read
else
#puts "kill r2"
fdes.delete r2
thread.kill if fdes == []
end
end
end
end
pid = Kernel.fork do
r1.close; r2.close
$defout.reopen(w1); $stdout.reopen(w1); $stderr.reopen(w2)
exec(command, *args)
# todo: should I care closing w1+w2 correctly?
end
w1.close; w2.close
status = Process.waitpid2(pid, 0)
# wait until EOF on both pipes (r1+r2)
thread.join
r1.close; r2.close

#$? = status[1]  # not possible
(status[1] == 0) ? true : false

end

There is also some unittesting of this ‘Kernel#system’ replacement:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/metaeditor/experiments/cpp_embed_ruby/test/output/testall.rb

Code for the sandbox:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/metaeditor/experiments/cpp_embed_ruby/test/output/b.rb

···

–
Simon Strandgaard

[Simon_Strandgaard]

Agree… thats the reason I want to use ‘select’…

How do I get data from the file-descriptor… and if no data is available
then the call should block until data becomes available ?

···

On Sun, 18 May 2003 01:50:00 +0900, Marcin ‘Qrczak’ Kowalczyk wrote:

If data is nil, then it’s EOF, so you should better not try to read further.
(If the input is a terminal, it will try to input more despite the user
hitting ^D, which is annoying.)

–
Simon Strandgaard

[Robert]

“Simon Strandgaard” 0bz63fz3m1qt3001@sneakemail.com schrieb im
Newsbeitrag news:pan.2003.05.19.01.12.43.69731@sneakemail.com…

replacement for ‘Kernel#system’ which captures all output.
This is the almost final version… this may have others interest
Warning: ‘stdin’ is not yet supported.

Suggestions for improvements is welcome.

Some minor (see lines without leading "> ") and a question: did you check
that this works on windows, too? I recently ran into a problem with pipes
on windows. Apparently they can’t be used for multiplexing…

def system(command, *args)
#todo: stdin
r1, w1 = IO.pipe # stdout
r2, w2 = IO.pipe # stderr
thread = Thread.new(r1,w1,r2,w2) do |r1,w1,r2,w2|
fdes = [r1, r2]
loop do
#puts “waiting”
res = select(fdes, nil, nil, nil)[0]
#p res
if res.include?(r1)
#puts “r1”
unless r1.eof
Redir.instance.write_out r1.read
else
#puts “kill r1”
fdes.delete r1
thread.kill if fdes ==
end
end
if res.include?(r2)
#puts “r2”
unless r2.eof
Redir.instance.write_error r2.read
else
#puts “kill r2”
fdes.delete r2
thread.kill if fdes ==
end
end
end
end
pid = Kernel.fork do
r1.close; r2.close
$defout.reopen(w1); $stdout.reopen(w1); $stderr.reopen(w2)
exec(command, *args)
# todo: should I care closing w1+w2 correctly?
end
w1.close; w2.close
status = Process.waitpid2(pid, 0)
# wait until EOF on both pipes (r1+r2)
thread.join
r1.close; r2.close

#$? = status[1]  # not possible

(status[1] == 0)

end

And I’d try to put the “close” calls into “ensure” blocks where possible.

Regards

robert

[Simon_Strandgaard]

did you check that this works on windows, too?
I recently ran into a problem with pipes on windows.
Apparently they can’t be used for multiplexing…

No, the code is untested on windows. I only have a FreeBSD box.
I am interested getting this code working on windows too.
feedback about windows would be nice.

Download these 2 files.

AEditor [Ruby BSD]

AEditor [Ruby BSD]

If it works the output should be similar to this.

ruby testall.rb

TestSandbox#test_fork pid=774
parent.leave
done
…
TestSandbox#test_system .
TestSandbox#test_system_args .
TestSandbox#test_system_status1 .
TestSandbox#test_system_status2 .
TestSandbox#test_system_status3 .
Time: 1.188965
OK (6/6 tests 17 asserts)

[snip code]

OK… I have added your changes to the code, thanks

And I’d try to put the “close” calls into “ensure” blocks where possible.

OK. I will try it out at sometime

···

On Mon, 19 May 2003 10:35:56 +0200, Robert Klemme wrote:

–
Simon Strandgaard

[Robert]

“Simon Strandgaard” 0bz63fz3m1qt3001@sneakemail.com schrieb im
Newsbeitrag news:pan.2003.05.19.09.29.28.72175@sneakemail.com…

did you check that this works on windows, too?
I recently ran into a problem with pipes on windows.
Apparently they can’t be used for multiplexing…

No, the code is untested on windows. I only have a FreeBSD box.
I am interested getting this code working on windows too.
feedback about windows would be nice.

Download these 2 files.

AEditor [Ruby BSD]

AEditor [Ruby BSD]

If it works the output should be similar to this.

Unfortunately not: it hangs in the first test “test_fork”.

ruby testall.rb

TestSandbox#test_fork pid=774
parent.leave
done
.
TestSandbox#test_system .
TestSandbox#test_system_args .
TestSandbox#test_system_status1 .
TestSandbox#test_system_status2 .
TestSandbox#test_system_status3 .
Time: 1.188965
OK (6/6 tests 17 asserts)

A question along the way: Why did you introduce a singleton class for
redirection? Why don’t you create an instance of a SandBox which receives
the redirecting streams and uses them when system is invoked? To me it
seems that a singleton is not in order because one might want to do
different redirections.

Regards

robert

···

On Mon, 19 May 2003 10:35:56 +0200, Robert Klemme wrote:

[Simon_Strandgaard]

If it works the output should be similar to this.

Unfortunately not: it hangs in the first test “test_fork”.

hmm. I don’t know what it takes to make it work on windows.
I think I saw something about ‘mkfifo’ but im not sure ?

A question along the way: Why did you introduce a singleton class for
redirection? Why don’t you create an instance of a SandBox which receives
the redirecting streams and uses them when system is invoked? To me it
seems that a singleton is not in order because one might want to do
different redirections.

make it work, make it right, make it fast
Right now im in the quick’n’dirty make $stdin work mode.
This is a great suggestion… I will add it to the todo list… thanks!

···

On Mon, 19 May 2003 16:40:51 +0200, Robert Klemme wrote:

–
Simon Strandgaard