"Dummy" IO object to push and pull data?

[Caleb_Clausen1 (Caleb Clausen)]

Ah, that's right. On windows, there's a special method you have to use
instead of select to check if input is available on a pipe. Let's see
..... I believe the method you're looking for is kbhit (or _kbhit
maybe? ms likes their underscores...) Check this page on MSDN:
  http://msdn.microsoft.com/en-us/library/58w7c94c(VS.80).aspx

There should be some library somewhere that makes this available to
you in rubyland.... Unfortunately, it does mean you have to poll the
pipe for available input instead of letting it notify you.

···

On 1/4/10, Shay Hawkins <gohegdeh@comcast.net> wrote:

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr. However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.

[Robert_K1 (Robert K.)]

That can't normally be, because the thread runs independently of the
loop. I am rather suspecting that you are having an issue with not
waiting for all threads to proper finish or do not make sure pipes are
read properly. You could also have created a dead lock that way, i.e.
by not reading all pipes properly because a process that tries to write
to a filled pipe is blocked.

I have created a pair of test scripts which I will place below. Strangely enough, there seems to be an issue with detecting closing of the stderr pipe which I need to research separately. If you just invoke the client script with "./clnt.rb 15 x" you will see the interaction properly working. If you, for example, increase the number and omit the reading of sin in the client the server will eventually block in one of the puts or printf statements. It may be that this is what you are observing.

Kind regards

  robert

Place both files in the same directory.

file clnt.rb:
#!/usr/local/bin/ruby19

# "client" which starts the server and every time a
# zero is read from the server's stderr pipe we send
# a notification to the server's stdin.
# server's stdout is used for logging state and is mirrored
# to this process's stdout with prefix "From server".

require 'open3'

# for debugging
Thread.abort_on_exception = true

Open3.popen3 File.join(File.dirname($0), "serv.rb"), *ARGV do |sin, sout, serr|
   sin.sync = true

   # we must make sure all pipes are either read or closed!
   r = Thread.new do
     sout.each do |line|
       puts "From server: #{line}"
     end
   end

   threads =

   serr.each do |line|
     printf "%-20s read %p\n", Time.now, line

     line.chomp!

     if line == "0"
       threads << Thread.new do
         sleep 5
         sin.puts "Received 0 five seconds ago."
       end
     elsif /finish/i =~ line
       # interestingly enough EOF detection does not
       # work with serv.rb
       break
     end
   end

   puts "finished reading"

   threads.each {|th| th.join}

   puts "finished notifying"

   # init shutdown sequence
   sin.close
   r.join
end

file serv.rb:
#!/usr/local/bin/ruby19

# "server" process which writes a fixed amount of numbers
# to stderr and then closes stderr to indicate it's finished.
# first arg is no of repetitions
# second arg if present says indicate end with a particular
# message to stderr.

$stdout.sync = $stderr.sync = true

puts "started"

th = Thread.new do
   $stdin.each do |line|
     printf "%-30s read %p\n", Time.now, line
   end
end

rep = (ARGV.shift || 10).to_i
ind = ARGV.shift

rep.times do
   $stderr.puts rand(3)
   sleep 1
end

puts "finished writing"

# indicate we are done
$stderr.puts "Finish" if ind
$stderr.close
puts "stderr.closed? #{$stderr.closed?}"

puts "finishing"

th.join

puts "finished"

···

On 04.01.2010 14:38, Shay Hawkins wrote:

Brian Candler wrote:

Sorry, I've read that several times and I still don't understand. Can you post a small example which demonstrates the problem, preferably standalone?

My apologies: I'll see if I can elaborate a bit...

Well, I originally was trying to find a workaround to this, but it seems the root of the problem is in the process' streams. I have a seperate program that I did not create myself (and therefore cannot change the code of) - this program puts output into stderr, and accepts input through stdin.

I call Open3.popen3 to create the three streams. In a loop, I want to constantly read input from stderr. (Assuming I assigned the pipes to stdin, stdout, and stderr variable names.)

loop do
   line = stderr.gets()
   puts("Output: #{line}")
end

Then, if I receive an output of say, 0, I want to input something to the program through stdin five seconds later.

loop do
   line = stderr.gets().strip()
   puts("Output: #{line}")

   if(line == "0")
      Thread.new() do
         sleep(5)
         stdin.puts("Received 0 five seconds ago.")
      end
   end
end

It creates the thread properly, but when the sleep expires and it is ready to call stdin.puts(), it cannot do anything, because the loop continued around and it is reading from stderr.

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[Shay_Hawkins]

Brian Candler wrote:

Perhaps it would be helpful to distinguish symptoms from your assumed
causes.

Firstly, are you saying that the stdin.puts line never completes? You
need to prove this. I suggest:

    STDERR.puts "About to send..."
    stdin.puts "Received 0"
    STDERR.puts "Finished sending."

Thirdly, because of buffering, the stuff you wrote to stdin might not
actually be sent until you do

    stdin.flush

I've actually tried these myself; it doesn't get past the stdin.puts
line, but rather only outputs the first line (where I'm using puts(),
not STDERR.puts()). As for flush, I've tried calling it in a few
different places - before the stdin.puts, after the stdin.puts,
inbetween loops, but it doesn't seem to do anything.

I'll look into the other things you mentioned and post my results.

Brian Candler wrote:

Of course, if it weren't for the "5 second delay" requirement, then it
could be written quite happily without a thread:

This is true; it works fine without a delay. The delay, unfortunately,
is the crucial aspect that I've been trying to get to work from the
beginning.

···

--
Posted via http://www.ruby-forum.com/\.

[Caleb_Clausen1 (Caleb Clausen)]

Except when I actually read the msdn link above, I see it only works
with stdin... that may not help you. I think there might be some other
windows-specific call you can make that checks any fd for input
availability... you might have to hunt around to find it, or ask on a
windows programming list.

···

On 1/4/10, Caleb Clausen <vikkous@gmail.com> wrote:

On 1/4/10, Shay Hawkins <gohegdeh@comcast.net> wrote:

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr. However, when it
receives a new set of input from stderr, since the stream is momentarily
being unused, it can then grab the stdin stream and properly input the
old line.

Ah, that's right. On windows, there's a special method you have to use
instead of select to check if input is available on a pipe. Let's see
..... I believe the method you're looking for is kbhit (or _kbhit
maybe? ms likes their underscores...) Check this page on MSDN:
  http://msdn.microsoft.com/en-us/library/58w7c94c(VS.80).aspx

There should be some library somewhere that makes this available to
you in rubyland.... Unfortunately, it does mean you have to poll the
pipe for available input instead of letting it notify you.

[Brian_Candler]

Robert Klemme wrote:

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr.

That can't normally be, because the thread runs independently of the
loop.

He's right, and I have replicated the problem. It occurs with with 1.8.6
one-click installer under Windows. With this interpreter, threads are
borked: when one thread is waiting on I/O, it prevents all other threads
from running.

The "official" 1.8.7 Windows build is OK, and so is a cygwin 1.8.7.

You can replicate the problem with this:

···

---------------------------------------
Thread.new do
  while line = $stdin.gets
    $stderr.puts ">>> Received #{line.chomp} <<<"
  end
end
i = -2
loop do
  $stderr.puts i
  i += 1
  sleep 2
end
---------------------------------------

Under the one-click installer 1.8.6, the printing loop freezes until you
give it some input.
--
Posted via http://www.ruby-forum.com/\.

[Shay_Hawkins]

Test program:

# Required support, go to process dir
require("open3.so")
Dir.chdir("..")

# Initiate the process and its streams
command = "********"
inline, outline, errline = Open3.popen3(command)

# Quick function for short timestamps
require("time")
def tstamp(line)
  puts("#{Time.now().to_s().split(" ")[3]} #{line}")
end

# Read from stderr, thread stdin input
loop do
  tstamp("> Awaiting data...")
  line = errline.gets().strip()
  tstamp("Output: #{line}")

  if(line == "0")
    tstamp("> Received the required line.")

    Thread.new() do
      tstamp("> The thread has been created.")
      sleep(5)

      tstamp("> Time's up! Sending input...")
      inline.puts("EXIT")
      tstamp("> The input has been sent.")
    end
  end
end

Output:

11:35:03 > Awaiting data...
11:35:03 Output: 2
11:35:03 > Awaiting data...
11:35:13 Output: 3
11:35:13 > Awaiting data...
11:35:23 Output: 0
11:35:23 > Received the required line.
11:35:23 > The thread has been created.
11:35:23 > Awaiting data...
11:35:33 Output: 4
11:35:33 > Awaiting data...
11:35:43 Output: 1
11:35:43 > Awaiting data...

···

--
Posted via http://www.ruby-forum.com/.

[Phillip_Gawlowski1 (Phillip Gawlowski)]

Robert Klemme wrote:

It creates the thread properly, but when the sleep expires and it is
ready to call stdin.puts(), it cannot do anything, because the loop
continued around and it is reading from stderr.

That can't normally be, because the thread runs independently of the
loop.

He's right, and I have replicated the problem. It occurs with with 1.8.6
one-click installer under Windows. With this interpreter, threads are
borked: when one thread is waiting on I/O, it prevents all other threads
from running.

The "official" 1.8.7 Windows build is OK, and so is a cygwin 1.8.7.

You can replicate the problem with this:

---------------------------------------
Thread.new do
   while line = $stdin.gets
     $stderr.puts ">>> Received #{line.chomp}<<<"
   end
end
i = -2
loop do
   $stderr.puts i
   i += 1
   sleep 2
end
---------------------------------------

Under the one-click installer 1.8.6, the printing loop freezes until you
give it some input.

For the heck of it, I've tested 1.8.6 and 1.9.1 compiled with MinGW:
PS C:\Scripts> ruby -v
ruby 1.9.1p243 (2009-07-16 revision 24175) [i386-mingw32]
PS C:\Scripts> c:\ruby\bin\ruby -v
ruby 1.8.6 (2009-08-04 patchlevel 383) [i386-mingw32]

Ruby 1.9.1 performs as expected: The threads start, and i gets incremented every 2 seconds:
PS C:\Scripts> ruby .\thread.rb
-2
-1
0
1
2
3
14
1
>>> Received 11 <<<
5
6
:q
>>> Received :q <<<
7
8

Same script with 1.8.6:
PS C:\Scripts> c:\ruby\bin\ruby .\thread.rb
-2

>>> Received <<<
1
>>> Received 1 <<<
-1
2
>>> Received 2 <<<
0
:q
>>> Received :q <<<
1
2

(Yeah, vim user here...)

It looks like 1.8.7 introduced (or rather got backported from 1.9) a change that makes threads work independent from the underlying OS.

···

On 05.01.2010 13:07, Brian Candler wrote:

--
Phillip Gawlowski

[Brian_Candler]

Shay Hawkins wrote:

Test program:

...

      tstamp("> Time's up! Sending input...")

But that message never appears in the output you showed - and this line
occurs *before* the puts. So at the moment looks either (a) sleep(5) is
sleeping indefinitely, or (b) the whole thread has terminated with an
exception.

A better test would be a standalone pair of ruby programs, because then
you have something which anyone can reproduce the problem with (*).

Here's your test program, very slightly modified, with a partner
program.

==> prg1.rb <==
Thread.abort_on_exception = true
require "open3"

command = "ruby prg2.rb"
inline, outline, errline = Open3.popen3(command)

def tstamp(line)
  puts("#{Time.now().to_s().split(" ")[3]} #{line}")
end

# Read from stderr, thread stdin input
loop do
  tstamp("> Awaiting data...")
  line = errline.gets().strip()
  tstamp("Output: #{line}")

  if(line == "0")
    tstamp("> Received the required line.")

    Thread.new() do
      tstamp("> The thread has been created.")
      sleep(5)

      tstamp("> Time's up! Sending input...")
      inline.puts("EXIT")
      tstamp("> The input has been sent.")
    end
  end
end

==> prg2.rb <==
i = -2
loop do
  $stderr.puts i
  i += 1
  if select([$stdin],nil,nil,2)
    line = $stdin.gets.chomp
    $stderr.puts ">>> Received #{line} <<<"
  end
end

And if I run it under Linux, here's what I get:

$ ruby prg1.rb
19:58:05 > Awaiting data...
19:58:05 Output: -2
19:58:05 > Awaiting data...
19:58:07 Output: -1
19:58:07 > Awaiting data...
19:58:09 Output: 0
19:58:09 > Received the required line.
19:58:09 > The thread has been created.
19:58:09 > Awaiting data...
19:58:11 Output: 1
19:58:11 > Awaiting data...
19:58:13 Output: 2
19:58:13 > Awaiting data...
19:58:14 > Time's up! Sending input...
19:58:14 > The input has been sent.
19:58:14 Output: >>> Received EXIT <<<
19:58:14 > Awaiting data...
19:58:14 Output: 3
19:58:14 > Awaiting data...
19:58:16 Output: 4
19:58:16 > Awaiting data...
19:58:18 Output: 5
19:58:18 > Awaiting data...
^Cprg1.rb:13:in `gets': Interrupt
  from prg1.rb:13
  from prg1.rb:11:in `loop'
  from prg1.rb:11

What happens if you run this under Windows?

Now, this slightly muddies the water because prg2.rb also depends on
being able to select(), as I wanted it to be able to sleep *and* receive
data on stdin, the same as your main program does. So it would be also
interesting to know what happens if you do "ruby prg2.rb" by itself. It
*should* print a new number every 2 seconds, and if you type something
onto stdin, it should echo it back as >>> Received ... <<<

What I'm trying to say is, if you can make prg2.rb work properly (or
make another of version of prg2 work using threads), then you ought to
be able to make prg1.rb work properly too.

(*) Incidentally: for someone to reproduce this problem, they'll need to
know exactly what platform you have. You said "Windows Vista" but you
didn't say which version of Ruby, nor which package (Cygwin? One-Click
Installer? Other?)

Regards,

Brian.

···

--
Posted via http://www.ruby-forum.com/\.

[Shay_Hawkins]

Brian Candler wrote:

What happens if you run this under Windows?

C:\Users\Shay\Desktop>ruby prg1.rb
15:51:20 > Awaiting data...
15:51:21 Output: -2
15:51:21 > Awaiting data...

Then nothing else happens. I'm 90% sure this is because Windows, being
the operating system that it is, holds up everything when .gets() is
called on stdin until something is provided.

Brian Candler wrote:

Now, this slightly muddies the water because prg2.rb also depends on
being able to select(), as I wanted it to be able to sleep *and* receive
data on stdin, the same as your main program does. So it would be also
interesting to know what happens if you do "ruby prg2.rb" by itself. It
*should* print a new number every 2 seconds, and if you type something
onto stdin, it should echo it back as >>> Received ... <<<

C:\Users\Shay\Desktop>ruby prg2.rb
-2
C:\Users\Shay\Desktop>Hello World!

Received Hello World! <<<

-1
C:\Users\Shay\Desktop>Blah~

Received Blah~ <<<

0

And so forth and so on.

Brian Candler wrote:

(*) Incidentally: for someone to reproduce this problem, they'll need to
know exactly what platform you have. You said "Windows Vista" but you
didn't say which version of Ruby, nor which package (Cygwin? One-Click
Installer? Other?)

C:\Users\Shay\Desktop>ruby -v
ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]

The program I used to install it (or at least what I have in my
downloads folder) is ruby186-27_rc2 from rubyinstaller.rubyforge.org.

···

--
Posted via http://www.ruby-forum.com/\.

[Brian_Candler]

Shay Hawkins wrote:

C:\Users\Shay\Desktop>ruby prg2.rb
-2
C:\Users\Shay\Desktop>Hello World!

Received Hello World! <<<

-1
C:\Users\Shay\Desktop>Blah~

Received Blah~ <<<

0

And so forth and so on.

It returns you to a command prompt? How odd.

What about a threaded version of prg2.rb, does it behave any
differently?

Thread.new do
  while line = $stdin.gets
    $stderr.puts ">>> Received #{line.chomp} <<<"
  end
end
i = -2
loop do
  $stderr.puts i
  i += 1
  sleep 2
end

···

--
Posted via http://www.ruby-forum.com/\.

[Robert_K1 (Robert K.)]

Actually part of that functionality is in Ruby itself because IO uses buffering by default. For tests like these it helps to place a line like this at the beginning of the script:

$stdout.sync = $stderr.sync = true

As far as I can see this wasn't done in Brian's test scripts so repeating with that line introduced will likely yield more accurate results.

Kind regards

  robert

···

On 01/04/2010 10:00 PM, Shay Hawkins wrote:

Brian Candler wrote:

What happens if you run this under Windows?

C:\Users\Shay\Desktop>ruby prg1.rb
15:51:20 > Awaiting data...
15:51:21 Output: -2
15:51:21 > Awaiting data...

Then nothing else happens. I'm 90% sure this is because Windows, being the operating system that it is, holds up everything when .gets() is called on stdin until something is provided.

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

[Shay_Hawkins]

Brian Candler wrote:

It returns you to a command prompt? How odd.

What about a threaded version of prg2.rb, does it behave any
differently?

Thread.new do
  while line = $stdin.gets
    $stderr.puts ">>> Received #{line.chomp} <<<"
  end
end
i = -2
loop do
  $stderr.puts i
  i += 1
  sleep 2
end

Oh, my apologies, I had put the user info there to show that I was
giving input (as if @echo hadn't been turned off). As for that program,
it behaves almost the same way - locking up until user gives input, then
displaying i if the 2 seconds have expired.

···

--
Posted via http://www.ruby-forum.com/\.

[Brian_Candler]

Robert Klemme wrote:

For tests like these it helps to place a line
like this at the beginning of the script:

$stdout.sync = $stderr.sync = true

I agree that's a good idea, but I don't think it actually makes a
difference here.

For one thing, the first output line ("-2") was shown on the terminal
immediately, so there's no reason why the second line ("-1") should not
be.

For another thing, the same program worked fine on the two 1.8.7
platforms but not the 1.8.6, on the same machine.

And finally, $stderr is usually unbuffered anyway (on Unix; I know I
can't make such assumptions about Windows).

···

--
Posted via http://www.ruby-forum.com/\.

[Brian_Candler]

Oh, my apologies, I had put the user info there to show that I was
giving input (as if @echo hadn't been turned off). As for that program,
it behaves almost the same way - locking up until user gives input, then
displaying i if the 2 seconds have expired.

OK. I've now got a Windows XP box beside me. I tried the threaded
version of prg2.rb with:

(1) rubyinstaller-1.8.6-p383-rc1.exe from
rubyforge.org/projects/rubyinstaller
(this is about a year newer than the 186-27_rc2 you have)

(2) cygwin-1.7.1 which has ruby-1.8.7-p72-2 package under 'interpreters'

(3) ruby 1.8.7-p72 binary from Download Ruby

The first fails in the way you described: that is, the sleeping thread
doesn't "wake up" until some input has been given to stdin.

However the second and third work just fine, in the same way as it does
under Linux. That is, I see a new number every 2 seconds, and I can type
in input whenever I like and get it echoed back immediately.

So clearly the problem is with how ruby is built in the one-click
installer. You may wish to raise a ticket against it, using this as a
demonstration of the problem. But in any case there are two other
working versions of ruby you can use instead.

HTH,

Brian.

···

--
Posted via http://www.ruby-forum.com/\.

[Shay_Hawkins]

Brian Candler wrote:

(3) ruby 1.8.7-p72 binary from Download Ruby

Ah, that'd be it. Everything works fine now - Thank you all for your
large amount of help.

···

--
Posted via http://www.ruby-forum.com/\.