Reading stdout & stderr from a pipe with popen3

ruby-talk
1

OK...using what ara.t.howard suggested, I've rewritten a block of my code and it looks something like this...

CODE

require "open3"
require "thread"

def runner(stdin, stdout, stderr, cmd)
  queue = Queue.new
  stdin.puts(cmd)
  errthd = Thread.new do
    Thread.current.abort_on_exception = true
    while(( line = stderr.gets ))
      queue.push(line)
    end
    queue.push :stderr_done
  end
  
  outthd = Thread.new do
    Thread.current.abort_on_exception = true
    while(( line = stdout.gets ))
      queue.push(line)
    end
    queue.push :stdout_done
  end
  
  inthd = Thread.new do
    Thread.current.abort_on_exception = true
    while(( stuff = queue.pop ))
      puts stuff
      break if :stdout_done and :stderr_done
    end
  end
  
  inthd.join
  errthd.exit
  outthd.exit
end

stdin, stdout, stderr = Open3.popen3("zmprov")
runner(stdin, stdout, stderr, "selectMailbox shares")
runner(stdin, stdout, stderr, "getAllFolders")
<<<<CODE

It almost works. I found that if I didn't add the errthd.exit and outthd.exit lines, the program would get stuck on inthd.join(waiting for the break conditions?) most of the time...very rarely would it complete. But the output is always incomplete. If I run the commands directly from the command line it look something like this...

GOOD OUTPUT

$ zmprov

selectMailbox shares

mailbox: shares@domain.com, size: 174.36 KB, messages: 115, unread: 59
mbox shares@domain.com> getAllFolders
        Id View Unread Msg Count Path
---------- ---- ---------- ---------- ----------
         1 conv 0 0 /
        16 docu 0 0 /Briefcase
        10 appo 0 0 /Calendar
        14 mess 0 0 /Chats
         7 cont 0 0 /Contacts
         6 mess 0 0 /Drafts
        13 cont 0 2 /Emailed Contacts
       257 appo 0 0 /evite
         2 mess 0 0 /Inbox
         4 mess 0 0 /Junk
        12 wiki 0 0 /Notebook
         5 mess 0 2 /Sent
        15 task 0 0 /Tasks
         3 conv 0 0 /Trash
<<<<GOOD OUTPUT

When I run my program though it looks more like this...

BAD OUTPUT

mailbox: shares@domain.com, size: 174.36 KB, messages: 115, unread: 59

mbox shares@domain.com> Id View Unread Msg Count Path
<<<<<BAD OUTPUT

The first line always seems to come out right... but the 'getAllFolders' command never comes out correctly. Maybe 1 in 10 times it will return a line or two more than just the 'mbox' line, but most of the time it's just that line of the output...the first line returned from the 'getAllFolders' command. So the thread for standard out doesn't seem to be grabbing all the data out of the pipe? I tried adding a short pause 'sleep 2' above the threads to see if maybe there was some slow response from them getting in the way, but that didn't really change anything.

Any help or ideas anyone can offer?

Thanks,
Matt

···

----- "ara.t.howard" <ara.t.howard@gmail.com> wrote:

the issue is even worse than you describe, the program can easily
become blocked if it's stdout or stderr pipes get full - to fix the
situation you need to use threads, one processing both stdout and
stderr asynchronously where each may have to trigger actions on
stdin. the general pattern is

   q = Queue.new

   err = Thread.new do
     Thread.current.abort_on_exception = true

     while(( line = stderr.gets ))
       ...
       q.push :somthing if some_condition_on(line)
     end
     q.push :stderr_done
   end

   out = Thread.new do
     Thread.current.abort_on_exception = true

     while(( line = stdout.gets ))
       ...
       q.push :something if some_condition_on(line)
     end
     q.push :stdout_done
   end

   in = Thread.new do
     Thread.current.abort_on_exception

     while(( command = q.pop ))
       ...
       break if stdout_done and stderr_done
     end
   end

   in.join

so basically have one thread sending commands down stdin. start a
thread each for stdout and stderr, each doing their own processing, if

they encounter something which means input needs to be send push it
onto a queue to allow the stdin thread to do it on their behave. this

of course ignores exceptional conditions and coordination between the

stdout and stderr threads, but it's one approach.

the big conditions any solution needs to handle are having no output

on either stderr or stdout or being blocked on a write to either due a

full pipe, which is why this cannot work safely:

   loop do
     handle stdout.gets
     handle stderr.gets
   end

check out open4 and session for examples of using threads to process

both stdout and stderr concurrently.

   http://codeforpeople.com/lib/ruby/
   # gem install open4 session

cheers.

a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being

better. simply reflect on that.
h.h. the 14th dalai lama

2

def runner(stdin, stdout, stderr, cmd)
queue = Queue.new
stdin.puts(cmd)

stdin.flush

errthd = Thread.new do
   Thread.current.abort_on_exception = true
   while(( line = stderr.gets ))
     queue.push(line)
   end
   queue.push :stderr_done
end

outthd = Thread.new do
   Thread.current.abort_on_exception = true
   while(( line = stdout.gets ))
     queue.push(line)
   end
   queue.push :stdout_done
end

inthd = Thread.new do
   Thread.current.abort_on_exception = true
   while(( stuff = queue.pop ))
     puts stuff
     break if :stdout_done and :stderr_done

this test reads "break if true and true" !?!

you are testing truth on two symbols. you need to keep vars and set them when :stdout_done and :stderr_done are seen on the queue.

if stuff == :stdout_done
   stdout_done == true
   next
end

if stuff == :stderr_done
   stderr_done = true
   next
end

if stdout_done and stderr_done
end

etc...

btw - you are simply re-writing the open4 gem - why not use it?

   end
end

inthd.join
errthd.exit
outthd.exit
end

a @ http://codeforpeople.com/

···

On Jul 17, 2008, at 10:53 AM, Matt Mencel wrote:
--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama