Ruby redirection woes

ruby-talk
1

Hi,

I have made a class, called CommandRunner, that I have been experiencing
some unusual behaviour with. I have lived with it for around two months
now, hoping I will stumble across a solution eventually. But alas, no.
:frowning: I designed the class to provide the following functions:

    - runs other programs in a child process (with exec()),
    - allows each command-line argument to be explicitly passed in as
      individual arguments of the CommandRunner's constructor, and
    - provides IO streams to the parent for the child's standard out,
      standard error, and standard in streams.
    
As far as I know, the existing available functions (such as Kernel.popen
and Kernel.system) can provide these, but one that does them all does
not exist. For instance, popen gives you only one IO stream for reading
the standard in AND error, and writing standard in; not seperate IO
streams for standard in and error. It also doesn't let you specify
command-line arguments seperately (according to the documentation,
atleast). If one of your arguments have spaces, then I think you will
need to quote it or something - I'm not sure.

Basically, when I run it and redirect output from the script to the
shell, I get additional output from the child process being run by
CommandRunner, when it should not be there.

To better explain, run the command_runner_shell_test.rb (file attached)
as follows, to see the wackiness:

     ruby command\_runner\_shell\_test\.rb > output\.txt      cat output.txt
    Hello, world! 2 times!
    Hello, world! 2 times!
    Hello, world! 2 times!

It should only print two times...

Increase the loop in command_runner_shell_test.rb to five times, and
watch the excess prints grow (what seems to be roughly) exponentially.

Uncomment the second block in the command_runner_shell_test.rb script,
and see it all get jumbled. More wackiness. o_O

All the weird thread and waiting for the 'R' character, trickery is so
that I can guaranty that by the time I am calling the waitpid function,
the process and PID still exist. In other words, it is possible for the
child process to be so quick that it finishes executing before waitpid()
gets called). So on a sidenote, does anyone know a cleaner way of
performing interprocess communication? Like how do I tell the forked,
child process that the parent is ready and waiting.

Another problem is sometimes it just hangs at the 'exec' line. I see the
process and command line being run from 'ps aux', but it doesn't seem to
ever exit. But one problem at a time. I suspect this one might be linked
with the above. Something to do with the file streams not being ready, I
think?

Take a look, and have fun figuring it out. :stuck_out_tongue:

Regards,

Steven

command_runner.rb (2.75 KB)

command_runner_shell_test.rb (381 Bytes)

路路路

--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E steven.wong@nautronix.com.au
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.

2

Since I haven't got any responses yet. Should I forward this to
ruby-core? Or is the explanation too lengthy?

Steven

路路路

On Tue, Oct 26, 2004 at 01:56:38PM +0900, Steven Kah Hien Wong wrote:

Hi,

I have made a class, called CommandRunner, that I have been experiencing
some unusual behaviour with. I have lived with it for around two months
now, hoping I will stumble across a solution eventually. But alas, no.
:frowning: I designed the class to provide the following functions:

--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E steven.wong@nautronix.com.au
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.

3

To better explain, run the command_runner_shell_test.rb (file attached)
as follows, to see the wackiness:

    $ ruby command_runner_shell_test.rb > output.txt
    $ cat output.txt
    Hello, world! 2 times!
    Hello, world! 2 times!
    Hello, world! 2 times!

It should only print two times...

I don't think you said anything about the platform you are running under.
When I run your test under ruby 1.8.2p2 and FreeBSD 4.10, I get:

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
$

and uncommenting the section labelled "more wackiness", I get

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
Hello, world! 2 times!
Hello, world! 3 times!
Hello, world! 3 times!
Hello, world! 3 times!
$

which looks pretty good to me too.

If you are running under cygwin plus some Microsoft operating system, then
there is a quotation provided to me by the 'fortune' program which I think
is relevant:

路路路

--------------------------------------------------------------------------
  There's a long-standing bug relating to the x86 architecture that
  allows you to install Windows.
                  -- Matthew D. Fuller
--------------------------------------------------------------------------

:slight_smile:

Seriously, Windows is badly broken with regard to processes and pipes.

All the weird thread and waiting for the 'R' character, trickery is so
that I can guaranty that by the time I am calling the waitpid function,
the process and PID still exist. In other words, it is possible for the
child process to be so quick that it finishes executing before waitpid()
gets called). So on a sidenote, does anyone know a cleaner way of
performing interprocess communication? Like how do I tell the forked,
child process that the parent is ready and waiting.

Why is this a problem? You can call waitpid on the child pid, and if it has
already terminated, then it will still be in the process table (as a zombie)
waiting for you to reap its exit status. That's unless someone has messed
with SIGCHLD.

You can prove this for yourself with a simple test program:

  child = fork do
    exit 42 # die immediately
  end
  
  sleep 5
  
  Process.waitpid(child)
  p $?

When I run this under FreeBSD, I get a delay of 5 seconds followed by
#<Process::Status: pid=1365,exited(42)>

If you use ps to see what is happening during the sleep, what I see is:

brian 1386 0.0 0.4 2544 1956 p7 S 11:15AM 0:00.01 ruby forktest.rb
brian 0 0.0 0.0 0 0 p7 ZW - 0:00.00 (ruby)
                                      ^
                       the zombie, waiting for collection

And even if the process had gone completely, you'd just get an error from
waitpid saying that the child did not exist (which is fine, since you know
it did exist at the time you forked, so it must have died since then). Only
if another process on the system had forked and re-used the same PID in the
mean time would that be a problem. Sensible Unixes are fine for this because
they allocate PIDs sequentially, and it's likely to be several minutes
before the same PID is re-used; however OpenBSD is completely broken, and
*could* re-use a PID within a microsecond of the previous child dying.

Regards,

Brian.

4

Hi,

路路路

In message "Re: Ruby redirection woes" on Mon, 1 Nov 2004 18:32:49 +0900, Steven Kah Hien Wong <steven.wong@nautronix.com.au> writes:

On Tue, Oct 26, 2004 at 01:56:38PM +0900, Steven Kah Hien Wong wrote:

Hi,

I have made a class, called CommandRunner, that I have been experiencing
some unusual behaviour with. I have lived with it for around two months
now, hoping I will stumble across a solution eventually. But alas, no.
:frowning: I designed the class to provide the following functions:

Since I haven't got any responses yet. Should I forward this to
ruby-core? Or is the explanation too lengthy?

Somehow I missed your first mail. I will check it later.
Stay tuned.

              matz.

5

Oops, copy-paste error. I actually get:

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
Hello, world! 2 times!
$

路路路

On Mon, Nov 01, 2004 at 11:18:58AM +0000, Brian Candler wrote:

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
$

6

Hi Brian,

Firstly, thanks for the help.

I don't think you said anything about the platform you are running under.
When I run your test under ruby 1.8.2p2 and FreeBSD 4.10, I get:

$ ruby command_runner_shell_test.rb
Hello, world! 2 times!
$

which looks pretty good to me too.

You actually need to run the test script with a shell redirection. Like
in the example output I gave, I redirected output of the script to a
file, output.txt, and then cat'ed it:

    $ ruby command_runner_shell_test.rb > output.txt
    $ cat output.txt
    Hello, world! 2 times!
    Hello, world! 2 times!
    Hello, world! 2 times!

Hopefully that will reproduce the same results I am seeing? Otherwise if
this problem just happens on mine... well that is just going to make it
even more difficult for me to track. :frowning:

If you are running under cygwin plus some Microsoft operating system

I am running this on Debian Linux 3.1, the testing branch of the
distribution. Ruby v1.8.2. Maybe the Debian package maintainers applied
some patches, but I doubt that would be the cause... but maybe.

Seriously, Windows is badly broken with regard to processes and pipes.

I would agree there. I once had to write a sh-like shell that compiled
and ran on both Windows and Linux, for a uni project. :slight_smile: I never
finished the windows part by the deadline (I did start a bit last minute
though)...

Why is this a problem? You can call waitpid on the child pid, and if it has
already terminated, then it will still be in the process table (as a zombie)
waiting for you to reap its exit status. That's unless someone has messed
with SIGCHLD.

That is just the handy little fact I needed. Now I guess I can remove
all that part of the code and be confident it works. Thanks!

And even if the process had gone completely, you'd just get an error from
waitpid saying that the child did not exist (which is fine, since you know
it did exist at the time you forked, so it must have died since then). Only
if another process on the system had forked and re-used the same PID in the
mean time would that be a problem.

Which was the (unlikely) problem I was trying to protect myself from. I
know it won't happen... but the fact that it "technically" could made me
uncomfortable. :stuck_out_tongue:

I've attached the 'updated' command_runner with the threading stuff
removed. Hopefully with that cruft gone, it is easier to read.

Steven

command_runner.rb (2.06 KB)

路路路

--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E steven.wong@nautronix.com.au
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.

7

Hi,

You actually need to run the test script with a shell redirection. Like
in the example output I gave, I redirected output of the script to a
file, output.txt, and then cat'ed it:

   $ ruby command_runner_shell_test.rb > output.txt
   $ cat output.txt
   Hello, world! 2 times!
   Hello, world! 2 times!
   Hello, world! 2 times!

Hopefully that will reproduce the same results I am seeing? Otherwise if
this problem just happens on mine... well that is just going to make it
even more difficult for me to track. :frowning:

Both parent and child processes flushes remaining output buffer. We
have to flush stdout and stderr before fork(2). Here's the patch.

              matz.
--- process.c 2 Oct 2004 03:50:48 -0000 1.92.2.11
+++ process.c 2 Nov 2004 07:07:49 -0000
@@ -1259,2 +1259,8 @@ rb_f_fork(obj)
     rb_secure(2);

路路路

In message "Re: Ruby redirection woes" on Tue, 2 Nov 2004 11:14:22 +0900, Steven Kah Hien Wong <steven.wong@nautronix.com.au> writes:
+
+#ifndef __VMS
+ fflush(stdout);
+ fflush(stderr);
+#endif
+
     switch (pid = fork()) {

8

You actually need to run the test script with a shell redirection. Like
in the example output I gave, I redirected output of the script to a
file, output.txt, and then cat'ed it:

    $ ruby command_runner_shell_test.rb > output.txt
    $ cat output.txt

My apologies for not reading your post more carefully. Yes, I get the same
here - and Matz has posted a patch which fixes it in Ruby. (However I'm not
sure that fix is necessarily good; the same problem would happen with other
filehandles which are open, and there's no proposal that fork() should flush
all those too).

However you can also just fix your command_runner.rb like this:

--- command_runner.rb.orig Tue Nov 2 13:21:01 2004
+++ command_runner.rb Tue Nov 2 13:21:41 2004
@@ -33,6 +33,8 @@
         child_to_parent_read, child_to_parent_write = IO.pipe
         child_to_parent_error_read, child_to_parent_error_write = IO.pipe
         
+ $stdout.flush
+ $stderr.flush
         @childPid = fork do
             parent_to_child_write.close
             child_to_parent_read.close

(The $stderr.flush maybe isn't needed, as I seem to remember that stderr
is unbuffered normally, but it won't do any harm to have it there)

Regards,

Brian.

9

Hi Matz,

That was fast. Just tested it on my box with Ruby-1.8.2-preview2, and it
seems to work:

$ ~/local/bin/ruby command_runner_shell_test.rb > out.txt
$ cat out.txt
Hello, world! 2 times!
Hello, world! 2 times!

All good. :slight_smile:

That's one less thing to scratch off my agenda of things to do.

Thank you! :slight_smile:

Steven

P.S. I am guessing this will be merged into the next release?

路路路

On Tue, Nov 02, 2004 at 04:08:05PM +0900, Yukihiro Matsumoto wrote:

Both parent and child processes flushes remaining output buffer. We
have to flush stdout and stderr before fork(2). Here's the patch.

--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E steven.wong@nautronix.com.au
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.

10

Yes, I guess you have a point there. I guess fork should be left as it
was before, then? Perhaps add documentation to flush stdout before
forking in the Ruby documentation somewhere.

Never knew forking could get so complicated. :frowning:

Steven

路路路

On Tue, Nov 02, 2004 at 10:25:51PM +0900, Brian Candler wrote:

(However I'm not sure that fix is necessarily good; the same problem
would happen with other filehandles which are open, and there's no
proposal that fork() should flush all those too).

--
NAUTRONIX LTD
Marine Technology Solutions

Steven Wong
Undergraduate Software Engineer

Nautronix Ltd ABN 28 009 019 603
108 Marine Terrace, Fremantle, WA 6160, Australia
T +61 (0)8 9431 0000, F +61 (0)8 9430 5901, http://www.nautronix.com

E steven.wong@nautronix.com.au
T +61 (0)8 9431 0024
M +61 (0)413 332 005

--
This email is confidential and intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are solely those of the author and do not necessarily represent those of NAUTRONIX LTD.

If you are not the intended recipient, you have received this email in error and use, dissemination, forwarding, printing, or copying of this email is strictly prohibited. If you have received this email in error please contact the sender.

Although our computer systems use active virus protection software, and we take various measures to reduce the risk of viruses being transmitted in e-mail messages and attachments sent from this company, we cannot guarantee that such e-mail messages and attachments are free from viruses on receipt. It is a condition of our using e-mail to correspond with you, that any and all liability on our part arising directly or indirectly out of any virus is excluded. Please ensure that you run virus checking software on all e-mail messages and attachments before reading them.

11

Hi,

路路路

In message "Re: Ruby redirection woes" on Tue, 2 Nov 2004 16:29:38 +0900, Steven Kah Hien Wong <steven.wong@nautronix.com.au> writes:

P.S. I am guessing this will be merged into the next release?

Yes. It already checked in to the stable CVS.

              matz.