Reading stdout & stderr from a pipe with popen3

ruby-talk
1

btw - you are simply re-writing the open4 gem - why not use it?

Because I already kind of knew Open3 and it took me long enough to figure out how to use that one....and I like to make my life as difficult as possible. :slight_smile:

So now I'm trying Open4 as you suggest and have a couple questions.

require "rubygems"
require "open4"

pid, stdin, stdout, stderr = Open4::popen4 "zmprov"
stdin.puts "sm shares"
stdin.puts "gaf"
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
exit

The check for stderr being empty? doesn't work...it prints a blank line if no error was returned so I tried checking for newline('\n') but it doesn't work either. What do I check for if nothing was returned on stderr?

The other thing I noticed is that I have to include the stdin.close or the program stops. I guess it's waiting for the close? It can't be after I try to print stderr/stdout either...it has to be before I check for those. So I guess I have to send all my input and then close the input and then check the statuses of stdout/stderr to decide how to proceed?

Looks like this will work and there doesn't seem to be any blocking getting in the way. Eventually I'm going to be sending thousands of these commands to the zmprov> prompt so I want this to be a tight and fast as possible. As long as I can get the full stdout/stdin of the last command before moving on to the next account...I think this will work.

Thanks,
Matt

2

Well drat....

pid, stdin, stdout, stderr = Open4::popen4 "zmprov"
stdin.puts "sm shares"
stdin.puts "gaf"
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
stdin.puts "sm shares2" # line 52
stdin.puts "gaf"
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
exit

Trying to do that doesn't work, it says the stream is closed and I can't send anything new to it.
./pipe_test.rb:52:in `write': closed stream (IOError)

So I don't want to stdin.close until I'm done processing a bunch of accounts, but I need to check stdout/stderr after I check each account without the stdin.close. Is that possible?

Matt

路路路

----- Original Message -----
From: "Matt Mencel" <MR-Mencel@wiu.edu>
To: "ruby-talk ML" <ruby-talk@ruby-lang.org>
Sent: Thursday, July 17, 2008 1:13:40 PM GMT -06:00 US/Canada Central
Subject: Re: Reading stdout & stderr from a pipe with popen3

btw - you are simply re-writing the open4 gem - why not use it?

Because I already kind of knew Open3 and it took me long enough to figure out how to use that one....and I like to make my life as difficult as possible. :slight_smile:

So now I'm trying Open4 as you suggest and have a couple questions.

require "rubygems"
require "open4"

pid, stdin, stdout, stderr = Open4::popen4 "zmprov"
stdin.puts "sm shares"
stdin.puts "gaf"
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
exit

The check for stderr being empty? doesn't work...it prints a blank line if no error was returned so I tried checking for newline('\n') but it doesn't work either. What do I check for if nothing was returned on stderr?

The other thing I noticed is that I have to include the stdin.close or the program stops. I guess it's waiting for the close? It can't be after I try to print stderr/stdout either...it has to be before I check for those. So I guess I have to send all my input and then close the input and then check the statuses of stdout/stderr to decide how to proceed?

Looks like this will work and there doesn't seem to be any blocking getting in the way. Eventually I'm going to be sending thousands of these commands to the zmprov> prompt so I want this to be a tight and fast as possible. As long as I can get the full stdout/stdin of the last command before moving on to the next account...I think this will work.

Thanks,
Matt

3

it's entirely up to the client program - this has nothing to do with open4 or any other program.

you've got all sorts if issues happening here

- you simply need to flush stdin, not close it, if you plan to continue to use it
- you are reading to the end of stdout and stderr, this blocks until a program exits, which will not happen until you close stdin, which is why you are needing to do that to see any output
- zmprov may or may not buffer stdout and stderr, i have no idea how this program is written an so cannot say

basically, if we assume zmprov is written in a line oriented fashion you *might* be able to do something like

   stdin.puts input

   output = stdout.gets output

   if output =~ /error/
    ....

but i see you are checking stderr too, so you have a problem: you have to read from both stdout and stderr and one, or both, might be empty. you program is going to block until you check both. so you have options

- use threads and a queue to check both, develop your own protocol using the code from open4 as an example of how to read from two io objects simultaneously
- use select to read from either stdout or stderr, whichever becomes available
- re-write zmprov so send all output on stderr and check only that

in short your question is impossible to answer without understanding what, *exactly* zmprov does and how it produces output on stdout and stderr. it's also impossible to solve without a solid understanding of how io and, specifically io buffering works.

hope that helps.

a @ http://codeforpeople.com/

路路路

On Jul 17, 2008, at 12:22 PM, Matt Mencel wrote:

Well drat....

pid, stdin, stdout, stderr = Open4::popen4 "zmprov"
stdin.puts "sm shares"
stdin.puts "gaf"
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
stdin.puts "sm shares2" # line 52
stdin.puts "gaf"
stdin.close
puts stderr.read.strip if !stderr.read.strip.empty?
puts stdout.read.strip if stderr.read.strip.empty?
exit

Trying to do that doesn't work, it says the stream is closed and I can't send anything new to it.
./pipe_test.rb:52:in `write': closed stream (IOError)

So I don't want to stdin.close until I'm done processing a bunch of accounts, but I need to check stdout/stderr after I check each account without the stdin.close. Is that possible?

Matt

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama