# Shell pipeline in Ruby? **URL:** \ **Category:** ruby-talk\ **Created:** [12 May 2011 16:05 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696 "2011-05-12T16:05:59Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [12 May 2011 16:05 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/1 "2011-05-12T16:05:59Z")
Hello, how do you write an equivalent of $ cmdA | cmdB | cmdC in Ruby? Specifically, I would like to see the PID, return value and stderr of each of these commands but I would like cmdB to read the stdout of cmdA directly, on its stdin, and similarily for cmdC and cmdB. Here I am not interested in feeding cmdA some particular input (eg. it can read /dev/null for all I care) but in general gluing arbitrary fd to its stdin might be desirable in other cases. Thanks Michal ---
**Author:** ![Jos\_Backus1](https://avatars.discourse-cdn.com/v4/letter/j/7feea3/32.png) [@Jos\_Backus1](https://rubytalk.org/u/Jos_Backus1)\ **Post date:** [12 May 2011 17:22 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/2 "2011-05-12T17:22:58Z")
Have you checked out Open3.pipeline? Jos > **···** > > -- > Peace cannot be achieved through violence, it can only be attained through > understanding. > On May 12, 2011 9:06 AM, "Michal Suchanek" \ wrote: > > > Hello, > > > > how do you write an equivalent of > > > > $ cmdA | cmdB | cmdC > > > > in Ruby? > > > > Specifically, I would like to see the PID, return value and stderr of > > each of these commands but I would like cmdB to read the stdout of > > cmdA directly, on its stdin, and similarily for cmdC and cmdB. > > > > Here I am not interested in feeding cmdA some particular input (eg. it > > can read /dev/null for all I care) but in general gluing arbitrary fd > > to its stdin might be desirable in other cases. > > > > Thanks > > > > Michal ---
**Author:** ![7stud](https://avatars.discourse-cdn.com/v4/letter/7/57b2e6/32.png) [@7stud](https://rubytalk.org/u/7stud)\ **Post date:** [12 May 2011 18:06 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/3 "2011-05-12T18:06:37Z")
Michal Suchanek wrote in post #998275: > Hello, > > how do you write an equivalent of > > $ cmdA | cmdB | cmdC How about: puts `cmdA | cmdB | cmdC` or puts %x{cmdA | cmdB | cmdC} > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Roger\_Pack4](https://avatars.discourse-cdn.com/v4/letter/r/bb73d2/32.png) [@Roger\_Pack4](https://rubytalk.org/u/Roger_Pack4)\ **Post date:** [13 May 2011 03:13 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/4 "2011-05-13T03:13:56Z")
In the spirit of the recent call for better ruby documentation, I wrote a writeup: [http://en.wikibooks.org/w/index.php?title=Ruby\_Programming/Running\_Multiple\_Processes&stable=0#Chaining\_processes](http://en.wikibooks.org/w/index.php?title=Ruby_Programming/Running_Multiple_Processes&stable=0#Chaining_processes) Enjoy. -roger- > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\ **Post date:** [13 May 2011 08:44 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/5 "2011-05-13T08:44:39Z")
$ ri Open3.pipeline\_r Open3.pipeline\_start Cheers robert > **···** > > On Thu, May 12, 2011 at 6:05 PM, Michal Suchanek \ wrote: > > > how do you write an equivalent of > > > > $ cmdA | cmdB | cmdC > > > > in Ruby? > > > > Specifically, I would like to see the PID, return value and stderr of > > each of these commands but I would like cmdB to read the stdout of > > cmdA directly, on its stdin, and similarily for cmdC and cmdB. > > > > Here I am not interested in feeding cmdA some particular input (eg. it > > can read /dev/null for all I care) but in general gluing arbitrary fd > > to its stdin might be desirable in other cases. > > -- > remember.guy do |as, often| as.you\_can - without end > [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/) ---
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [13 May 2011 15:32 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/6 "2011-05-13T15:32:22Z")
OK, thanks for all the replies. It seems this is not doable with ruby 1.8 but spawn and similar methods in 1.9 grew the named arguments which include the file descriptors. Here is a working (although not polished) solution for ruby 1.9. First a short testing script called cmd: #!/usr/bin/ruby prefix = ARGV[0] + ":" append = ARGV[1] while l = STDIN.gets do   STDOUT \<\< l   STDERR \<\< prefix \<\< l end STDOUT.puts append And the long script with pipes that calls the above: #!/usr/bin/ruby1.9.1 # Debian calls ruby 1.9 ruby1.9.1 pipes = (1..4).each {pipes \<\< IO.pipe} errs = (1..3).each {errs \<\< IO.pipe} writer = pipes[0][1] reader = pipes[3][0] i = 1 # Careful about pipe direction, it only works one way. pids = [ spawn("cmd", "a", "aa", :in =\> pipes[0][0], :out =\> pipes[1][1], :err =\> errs[0][1] ), spawn("cmd", "b", "bb", :in =\> pipes[1][0], :out =\> pipes[2][1], :err =\> errs[1][1] ), spawn("cmd", "c", "cc", :in =\> pipes[2][0], :out =\> pipes[3][1], :err =\> errs[2][1] ), ] # We don't really want to see these pipes, perhaps some Open3.pipeline\_\* method would create them out of sight pipes.flatten.select{|p| p != writer && p != reader}.each{|p|p.close}\ maxlen = 128 # some arbitrary size for chunk of data read at once outputs = errs.collect{|r,w|r} outputs \<\< reader wsel = [writer] while true do   ios = IO.select(outputs, wsel, nil, 1)   STDERR \<\< '.'     string = "%4i\n" % i     if writer then       begin         result = writer.write\_nonblock(string)         i = i + 1         if ( i \> 1000 ) then           writer.close           writer = nil           wsel = nil         end       rescue IO::WaitWritable, Errno::EINTR       end     end   outputs.each{|io|     begin       result = io.read\_nonblock(maxlen)       STDOUT \<\< result     rescue IO::WaitReadable, Errno::EINTR, EOFError       if $!.is\_a? EOFError then         STDERR.puts "EOF"         outputs.reject!{|o| o == io }       end     end   }   (0...3).each{|i| p = pids[i]     if p and Process.waitpid(p, Process::WNOHANG) then       STDOUT.puts $?.inspect       pids[i] = nil     end   }   break if !ios && pids.select{|pid| pid} == end > **···** > > On 12 May 2011 18:05, Michal Suchanek \ wrote: > > > Hello, > > > > how do you write an equivalent of > > > > $ cmdA | cmdB | cmdC > > > > in Ruby? > > > > Specifically, I would like to see the PID, return value and stderr of > > each of these commands but I would like cmdB to read the stdout of > > cmdA directly, on its stdin, and similarily for cmdC and cmdB. > > > > Here I am not interested in feeding cmdA some particular input (eg. it > > can read /dev/null for all I care) but in general gluing arbitrary fd > > to its stdin might be desirable in other cases. > > > > Thanks > > > > Michal ---
**Author:** ![Brian\_Candler](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@Brian\_Candler](https://rubytalk.org/u/Brian_Candler)\ **Post date:** [14 May 2011 15:14 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/7 "2011-05-14T15:14:50Z")
Here is a ruby 1.9.2 version with close\_on\_exec=. I leave it as an exercise to refactor this to an arbitrary pipeline of N commands - it would return one input pipe, one output pipe, N pids and N error pipes. Regards, Brian. #Thread.abort\_on\_exception = true # for debugging ruby\_to\_a\_rd, ruby\_to\_a\_wr = IO.pipe ruby\_to\_a\_wr.close\_on\_exec = true # no children should have this a\_to\_b\_rd, a\_to\_b\_wr = IO.pipe a\_err\_rd, a\_err\_wr = IO.pipe a\_err\_rd.close\_on\_exec = true # no children should have this pid1 = fork do   a\_to\_b\_rd.close   STDIN.reopen(ruby\_to\_a\_rd)   STDOUT.reopen(a\_to\_b\_wr)   STDERR.reopen(a\_err\_wr)   exec("cat; echo done cat 1\>&2")   STDERR.puts "Whoops! #{$!}" end # we don't want these fds, nor any of the further children ruby\_to\_a\_rd.close a\_to\_b\_wr.close a\_err\_wr.close b\_to\_c\_rd, b\_to\_c\_wr = IO.pipe b\_err\_rd, b\_err\_wr = IO.pipe b\_err\_rd.close\_on\_exec = true # no children should have this pid2 = fork do   b\_to\_c\_rd.close   STDIN.reopen(a\_to\_b\_rd)   STDOUT.reopen(b\_to\_c\_wr)   STDERR.reopen(b\_err\_wr)   exec("tr 'a-z' 'A-Z'; echo done tr 1\>&2")   STDERR.puts "Whoops! #{$!}" end # we don't want these fds, nor any of the further children a\_to\_b\_rd.close b\_to\_c\_wr.close b\_err\_wr.close c\_to\_ruby\_rd, c\_to\_ruby\_wr = IO.pipe c\_err\_rd, c\_err\_wr = IO.pipe c\_err\_rd.close\_on\_exec = true # no children should have this pid3 = fork do   c\_to\_ruby\_rd.close   STDIN.reopen(b\_to\_c\_rd)   STDOUT.reopen(c\_to\_ruby\_wr)   STDERR.reopen(c\_err\_wr)   exec("sed 's/O/0/g'; echo done sed 1\>&2")   STDERR.puts "Whoops! #{$!}" end # we don't want these fds, nor any of the further children b\_to\_c\_rd.close c\_to\_ruby\_wr.close c\_err\_wr.close Thread.new do   ruby\_to\_a\_wr.puts "Here is some data"   ruby\_to\_a\_wr.puts "And some more"   ruby\_to\_a\_wr.close end Thread.new do   while line = a\_err\_rd.gets     puts "A err: #{line}"   end   a\_err\_rd.close end Thread.new do   while line = b\_err\_rd.gets     puts "B err: #{line}"   end   b\_err\_rd.close end Thread.new do   while line = c\_err\_rd.gets     puts "C err: #{line}"   end   c\_err\_rd.close end while line = c\_to\_ruby\_rd.gets   puts line end c\_to\_ruby\_rd.close > **···** > > -- > Posted via [http://www.ruby-forum.com/](http://www.ruby-forum.com/). ---
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [16 May 2011 15:29 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/8 "2011-05-16T15:29:48Z")
Thanks for all the replies. To sum up there are multiple methods to achieve this. - fork - works on ruby 1.8 but not on Windows - Open3 - very nice interface with pipeline\_start and such which does much of the setup behind the scenes but from what I read on the Net does not work on Windows, again - Open3-win32 and Open4 exist to address this issue but they may not have the same interface. - Process.spawn - works only in ruby 1.9 but should be portable to Windows should you need it. Basically equivalent to Open3 except you see all the gory details. Not sure why Open3 does not use this on Windows. Maybe Open3 was in 1.8 already when spawn did not allow for redirects or something. - Shell - somewhat not well known and not well documented Ruby library. You will reimplement this if you use one of the above 😉 For reading and writing numerous pipes you can use either a Ruby thread per pipe or a select loop, either works. Thanks Michal > **···** > > On 12 May 2011 18:05, Michal Suchanek \ wrote: > > > Hello, > > > > how do you write an equivalent of > > > > $ cmdA | cmdB | cmdC > > > > in Ruby? > > > > Specifically, I would like to see the PID, return value and stderr of > > each of these commands but I would like cmdB to read the stdout of > > cmdA directly, on its stdin, and similarily for cmdC and cmdB. > > > > Here I am not interested in feeding cmdA some particular input (eg. it > > can read /dev/null for all I care) but in general gluing arbitrary fd > > to its stdin might be desirable in other cases. ---
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [12 May 2011 19:41 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/9 "2011-05-12T19:41:17Z")
I have checked all the stuff I could find once but did not find a solution. That's why I am asking, perhaps I overlooked something. While the pipeline gives the status of all the involved processes and connects them properly which is more than I ever got from Ruby it is not obvious how to capture the output - the stdout of the last process can be captured with pipeline\_r but the stderrs get sinked somewhere in the pipeline\_run. Also the in and out options used in the example are not documented afaict. Thanks Michal > **···** > > On 12 May 2011 19:22, Jos Backus \ wrote: > > > Have you checked out Open3.pipeline? ---
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [13 May 2011 08:33 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/10 "2011-05-13T08:33:42Z")
> In the spirit of the recent call for better ruby documentation, I wrote Unfortunately, this does not work. First, these pipelines don't always work as one would expect so testing is \*required\* before posting a "solution". Second, afaict the reopen method only changes the IOs at Ruby level, not the fd:s at C level so it's useless in this context. Thanks Michal > **···** > > On 13 May 2011 05:13, Roger Pack \ wrote: > > > a > > writeup: > > > > [Ruby Programming/Running Multiple Processes - Wikibooks, open books for an open world](http://en.wikibooks.org/w/index.php?title=Ruby_Programming/Running_Multiple_Processes&stable=0#Chaining_processes) > > > > Enjoy. > > -roger- > > > > -- > > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Lionel](https://avatars.discourse-cdn.com/v4/letter/l/f9ae1b/32.png) [@Lionel](https://rubytalk.org/u/Lionel)\ **Post date:** [13 May 2011 09:35 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/11 "2011-05-13T09:35:34Z")
+1 for Robert's answer. From the Open3.pipeline\_start doc, an example is given: # convert pdf to ps and send it to a printer. # collect error message of pdftops and lpr. pdf\_file = "paper.pdf" printer = "printer-name" err\_r, err\_w = IO.pipe Open3.pipeline\_start(["pdftops", pdf\_file, "-"],                      ["lpr", "-P#{printer}"],                      :err=\>err\_w) {|ts|   err\_w.close   p err\_r.read # error messages of pdftops and lpr. } Cheers Lionel ---
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [13 May 2011 13:27 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/12 "2011-05-13T13:27:26Z")
Again, like in Open3.pipeline the :err argument used in the example is not documented. From the description of the example it seems it captures the stderr of all the commands together. Are there more arguments like that? What do they mean? Also this is available in ruby 1.9 only. Thanks Michal > **···** > > On 13 May 2011 10:44, Robert Klemme \ wrote: > > > On Thu, May 12, 2011 at 6:05 PM, Michal Suchanek \ wrote: > > > > > how do you write an equivalent of > > > > > > $ cmdA | cmdB | cmdC > > > > > > in Ruby? > > > > > > Specifically, I would like to see the PID, return value and stderr of > > > each of these commands but I would like cmdB to read the stdout of > > > cmdA directly, on its stdin, and similarily for cmdC and cmdB. > > > > > > Here I am not interested in feeding cmdA some particular input (eg. it > > > can read /dev/null for all I care) but in general gluing arbitrary fd > > > to its stdin might be desirable in other cases. > > > > $ ri Open3.pipeline\_r Open3.pipeline\_start ---
**Author:** ![Roger\_Pack4](https://avatars.discourse-cdn.com/v4/letter/r/bb73d2/32.png) [@Roger\_Pack4](https://rubytalk.org/u/Roger_Pack4)\ **Post date:** [13 May 2011 13:48 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/13 "2011-05-13T13:48:27Z")
Roger Pack wrote in post #998434: > In the spirit of the recent call for better ruby documentation, I wrote > a > writeup: [http://en.wikibooks.org/w/index.php?title=Ruby\_Programming/Running\_Multiple\_Processes&stable=0#Chaining\_processes](http://en.wikibooks.org/w/index.php?title=Ruby_Programming/Running_Multiple_Processes&stable=0#Chaining_processes) Unfortunately it appears the wiki formatting is mangled in firefox, but works well in chrome 🙂 Here's the complete code snippet: pipe\_me\_in, pipe\_peer\_out = IO.pipe pipe\_peer\_in, pipe\_me\_out = IO.pipe fork do   STDIN.reopen(pipe\_peer\_in)   STDOUT.reopen(pipe\_peer\_out)   Kernel.exec("echo 33")   # this line is never executed because exec moves the process end pipe\_peer\_out.close # file handles have to all be closed in order for the "read" method, below, to be able to know # that it's done reading data, so it can return. See also > **[Beware of pipe duplication in subprocesses](https://devver.wordpress.com/2009/10/22/beware-of-pipe-duplication-in-subprocesses/)** > > I’ve been writing a lot lately about managing subprocesses in Ruby, and in that vein I thought I’d write up an interesting subprocess gotcha we encountered recently. I won’t go in… pipe\_me\_in.read There is a link in there to a blog describing ruby's shell class, perhaps you did not see it? -r > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Brian\_Candler](https://avatars.discourse-cdn.com/v4/letter/b/5f9b8f/32.png) [@Brian\_Candler](https://rubytalk.org/u/Brian_Candler)\ **Post date:** [14 May 2011 15:05 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/14 "2011-05-14T15:05:47Z")
Michal Suchanek wrote in post #998548: > And the long script with pipes that calls the above: Since you are doing this on a real operating system, there's no need to use ruby to copy the output of one process to the input of the next process (unless you want to capture the communication between the processes as well). OTOH, making sure all the right FDs are closed in each child can be tricky in 1.8, because there's no IO.close\_on\_exec= (although 1.9 has this). Here is an example which works in 1.8.7. Obviously it's possible to refactor this to N children, although I expect you'll end up with something like the 'pipeline' method mentioned before if you do. > **···** > > -------------------------------- > #Thread.abort\_on\_exception = true # for debugging > > ruby\_to\_a\_rd, ruby\_to\_a\_wr = IO.pipe > a\_to\_b\_rd, a\_to\_b\_wr = IO.pipe > a\_err\_rd, a\_err\_wr = IO.pipe > # open: r2a\_r, r2a\_w, a2b\_r, a2b\_w, ae\_r, ae\_w > pid1 = fork do >   ruby\_to\_a\_wr.close >   a\_to\_b\_rd.close >   a\_err\_rd.close >   STDIN.reopen(ruby\_to\_a\_rd) >   STDOUT.reopen(a\_to\_b\_wr) >   STDERR.reopen(a\_err\_wr) >   exec("cat; echo done cat 1\>&2") >   STDERR.puts "Whoops! #{$!}" > end > ruby\_to\_a\_rd.close > a\_to\_b\_wr.close > a\_err\_wr.close > # open: r2a\_w, a2b\_r, ae\_r > > b\_to\_c\_rd, b\_to\_c\_wr = IO.pipe > b\_err\_rd, b\_err\_wr = IO.pipe > # open: r2a\_w, a2b\_r, ae\_r, b2c\_r, b2c\_w, be\_r, be\_w > pid2 = fork do >   ruby\_to\_a\_wr.close >   b\_to\_c\_rd.close >   a\_err\_rd.close >   b\_err\_rd.close >   STDIN.reopen(a\_to\_b\_rd) >   STDOUT.reopen(b\_to\_c\_wr) >   STDERR.reopen(b\_err\_wr) >   exec("tr 'a-z' 'A-Z'; echo done tr 1\>&2") >   STDERR.puts "Whoops! #{$!}" > end > a\_to\_b\_rd.close > b\_to\_c\_wr.close > b\_err\_wr.close > # open: r2a\_w, ae\_r, b2c\_r, be\_r > > c\_to\_ruby\_rd, c\_to\_ruby\_wr = IO.pipe > c\_err\_rd, c\_err\_wr = IO.pipe > # open: r2a\_w, ae\_r, b2c\_r, be\_r, c2r\_r, c2r\_w, ce\_r, ce\_w > pid3 = fork do >   ruby\_to\_a\_wr.close >   c\_to\_ruby\_rd.close >   a\_err\_rd.close >   b\_err\_rd.close >   c\_err\_rd.close >   STDIN.reopen(b\_to\_c\_rd) >   STDOUT.reopen(c\_to\_ruby\_wr) >   STDERR.reopen(c\_err\_wr) >   exec("sed 's/O/0/g'; echo done sed 1\>&2") >   STDERR.puts "Whoops! #{$!}" > end > b\_to\_c\_rd.close > c\_to\_ruby\_wr.close > c\_err\_wr.close > # open: r2a\_w, ae\_r, be\_r, c2r\_r, ce\_r > > Thread.new do >   ruby\_to\_a\_wr.puts "Here is some data" >   ruby\_to\_a\_wr.puts "And some more" >   ruby\_to\_a\_wr.close > end > > Thread.new do >   while line = a\_err\_rd.gets >     puts "A err: #{line}" >   end >   a\_err\_rd.close > end > > Thread.new do >   while line = b\_err\_rd.gets >     puts "B err: #{line}" >   end >   b\_err\_rd.close > end > > Thread.new do >   while line = c\_err\_rd.gets >     puts "C err: #{line}" >   end >   c\_err\_rd.close > end > > while line = c\_to\_ruby\_rd.gets >   puts line > end > c\_to\_ruby\_rd.close > -------------------------------- > > This can be simplified a bit if you don't want three separate error > streams; they can all share the same one. i.e. each child would have > >   err\_rd.close >   STDERR.reopen(err\_wr) > > Regards, > > Brian. > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Stu1](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@Stu1](https://rubytalk.org/u/Stu1)\ **Post date:** [12 May 2011 19:54 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/15 "2011-05-12T19:54:52Z")
There is also popen4 which I believe also gives you the pid with std - in/out/err > **···** > > On Thu, May 12, 2011 at 2:41 PM, Michal Suchanek \ wrote: > > > On 12 May 2011 19:22, Jos Backus \ wrote: > > > > > Have you checked out Open3.pipeline? > > > > I have checked all the stuff I could find once but did not find a solution. > > > > That's why I am asking, perhaps I overlooked something. > > > > While the pipeline gives the status of all the involved processes and > > connects them properly which is more than I ever got from Ruby it is > > not obvious how to capture the output - the stdout of the last > > process can be captured with pipeline\_r but the stderrs get sinked > > somewhere in the pipeline\_run. > > > > Also the in and out options used in the example are not documented afaict. > > > > Thanks > > > > Michal ---
**Author:** ![Roger\_Pack4](https://avatars.discourse-cdn.com/v4/letter/r/bb73d2/32.png) [@Roger\_Pack4](https://rubytalk.org/u/Roger_Pack4)\ **Post date:** [13 May 2011 13:44 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/16 "2011-05-13T13:44:09Z")
> Unfortunately, this does not work. Try it now, fixed the bug. > Second, afaict the reopen method only changes the IOs at Ruby level, > not the fd:s at C level so it's useless in this context. Try it out. This is, AFAIK, the "standard" way to pipeline processes, and works in 1.8 even. GL! -r > **···** > > -- > Posted via [http://www.ruby-forum.com/\](http://www.ruby-forum.com/%5C). ---
**Author:** ![Robert\_K1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/robert_k1/32/1830_2.png) [@Robert\_K1](https://rubytalk.org/u/Robert_K1)\ **Post date:** [13 May 2011 14:23 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/17 "2011-05-13T14:23:15Z")
> > > how do you write an equivalent of > > > > > > $ cmdA | cmdB | cmdC > > > > > > in Ruby? > > > > > > Specifically, I would like to see the PID, return value and stderr of > > > each of these commands but I would like cmdB to read the stdout of > > > cmdA directly, on its stdin, and similarily for cmdC and cmdB. > > > > > > Here I am not interested in feeding cmdA some particular input (eg. it > > > can read /dev/null for all I care) but in general gluing arbitrary fd > > > to its stdin might be desirable in other cases. > > > > $ ri Open3.pipeline\_r Open3.pipeline\_start > > Again, like in Open3.pipeline the :err argument used in the example is > not documented. Not true. From ri Open3.pipeline\_r in 1.9.2p180: Each cmd is a string or an array. If it is an array, the elements are passed to Process.spawn.   cmd:     commandline command line string which is passed to a shell     [env, commandline, opts] command line string which is passed to a shell     [env, cmdname, arg1, ..., opts] command name and one or more arguments (no shell)     [env, [cmdname, argv0], arg1, ..., opts] command name and arguments including argv[0] (no shell)   Note that env and opts are optional, as Process.spawn. Then $ ri Process.spawn > From the description of the example it seems it captures the stderr of > all the commands together. > > Are there more arguments like that? > > What do they mean? See above. > Also this is available in ruby 1.9 only. Well? Cheers robert > **···** > > On Fri, May 13, 2011 at 3:27 PM, Michal Suchanek \ wrote: > > > On 13 May 2011 10:44, Robert Klemme \ wrote: > > > > > On Thu, May 12, 2011 at 6:05 PM, Michal Suchanek \ wrote: > > -- > remember.guy do |as, often| as.you\_can - without end > [http://blog.rubybestpractices.com/](http://blog.rubybestpractices.com/) ---
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [13 May 2011 18:14 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/18 "2011-05-13T18:14:34Z")
> Roger Pack wrote in post #998434: > > > In the spirit of the recent call for better ruby documentation, I wrote > > a > > writeup: > > [Ruby Programming/Running Multiple Processes - Wikibooks, open books for an open world](http://en.wikibooks.org/w/index.php?title=Ruby_Programming/Running_Multiple_Processes&stable=0#Chaining_processes) > > Unfortunately it appears the wiki formatting is mangled in firefox, but > works well in chrome 🙂 > > Here's the complete code snippet: > > pipe\_me\_in, pipe\_peer\_out = IO.pipe > pipe\_peer\_in, pipe\_me\_out = IO.pipe > > fork do > STDIN.reopen(pipe\_peer\_in) > STDOUT.reopen(pipe\_peer\_out) > Kernel.exec("echo 33") > # this line is never executed because exec moves the process > end > pipe\_peer\_out.close # file handles have to all be closed in order for > the "read" method, below, to be able to know > # that it's done reading data, so it can return. See also > [Beware of pipe duplication in subprocesses | The Devver Blog](http://devver.wordpress.com/2009/10/22/beware-of-pipe-duplication-in-subprocesses/) > pipe\_me\_in.read Changing the last line to puts '"' + pipe\_me\_in.read.chomp + '"' Really writes 33 quoted so IO.reopen works for setting up the file descriptors. It is not clear from the docs if it would work or not. It does work, though: out = File.open("testfile","w") STDOUT.reopen(out) system("echo 33") STDERR.puts '"' + File.read("testfile").chomp + '"' =\> "33" I vaguely recall some issues with using STDIN and STDOUT for redirection earlier but it might be due to using them incorrectly. > There is a link in there to a blog describing ruby's shell class, > perhaps you did not see it? Yes, I missed the link labeled just [1] Thanks Michal > **···** > > On 13 May 2011 15:48, Roger Pack \ wrote: ---
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [16 May 2011 11:05 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/19 "2011-05-16T11:05:40Z")
I don't use Ruby for that. And I used a select loop because I had issues with threads in the past. However, your example works perfectly for me, even when updated to put more data through each of the pipes to make sure they don't end up all in a buffer and the threads have to be actually switched for the pipeline to work. I think there might still be issues if the produced data was without line endings because none of the gets would finish but that can be solved by using read\_nonblock or somesuch instead. Thanks Michal > **···** > > On 14 May 2011 17:05, Brian Candler \ wrote: > > > Michal Suchanek wrote in post #998548: > > > > > And the long script with pipes that calls the above: > > > > Since you are doing this on a real operating system, there's no need to > > use ruby to copy the output of one process to the input of the next > > ----------- > #Thread.abort\_on\_exception = true # for debugging > > ruby\_to\_a\_rd, ruby\_to\_a\_wr = IO.pipe > a\_to\_b\_rd, a\_to\_b\_wr = IO.pipe > a\_err\_rd, a\_err\_wr = IO.pipe > # open: r2a\_r, r2a\_w, a2b\_r, a2b\_w, ae\_r, ae\_w > pid1 = fork do >     ruby\_to\_a\_wr.close >     a\_to\_b\_rd.close >     a\_err\_rd.close >     STDIN.reopen(ruby\_to\_a\_rd) >     STDOUT.reopen(a\_to\_b\_wr) >     STDERR.reopen(a\_err\_wr) >     exec('while read x ; do echo "$x" ; echo "$x" \>&2 ; done') >     STDERR.puts "Whoops! #{$!}" > end > ruby\_to\_a\_rd.close > a\_to\_b\_wr.close > a\_err\_wr.close > # open: r2a\_w, a2b\_r, ae\_r > > b\_to\_c\_rd, b\_to\_c\_wr = IO.pipe > b\_err\_rd, b\_err\_wr = IO.pipe > # open: r2a\_w, a2b\_r, ae\_r, b2c\_r, b2c\_w, be\_r, be\_w > pid2 = fork do >     ruby\_to\_a\_wr.close >     b\_to\_c\_rd.close >     a\_err\_rd.close >     b\_err\_rd.close >     STDIN.reopen(a\_to\_b\_rd) >     STDOUT.reopen(b\_to\_c\_wr) >     STDERR.reopen(b\_err\_wr) >     exec('while read x ; do x=`echo "$x" | tr a-z A-Z` ; echo "$x" ; > echo "$x" \>&2 ; done') >     STDERR.puts "Whoops! #{$!}" > end > a\_to\_b\_rd.close > b\_to\_c\_wr.close > b\_err\_wr.close > # open: r2a\_w, ae\_r, b2c\_r, be\_r > > c\_to\_ruby\_rd, c\_to\_ruby\_wr = IO.pipe > c\_err\_rd, c\_err\_wr = IO.pipe > # open: r2a\_w, ae\_r, b2c\_r, be\_r, c2r\_r, c2r\_w, ce\_r, ce\_w > pid3 = fork do >     ruby\_to\_a\_wr.close >     c\_to\_ruby\_rd.close >     a\_err\_rd.close >     b\_err\_rd.close >     c\_err\_rd.close >     STDIN.reopen(b\_to\_c\_rd) >     STDOUT.reopen(c\_to\_ruby\_wr) >     STDERR.reopen(c\_err\_wr) >     exec('while read x ; do x=`echo "$x" | sed -e s/O/0/g` ; echo "$x" > ; echo "$x" \>&2 ; done') >     STDERR.puts "Whoops! #{$!}" > end > b\_to\_c\_rd.close > c\_to\_ruby\_wr.close > c\_err\_wr.close > # open: r2a\_w, ae\_r, be\_r, c2r\_r, ce\_r > > Thread.new do >     ruby\_to\_a\_wr.puts "Here is some data" >     (1..1000).each{|i| >         ruby\_to\_a\_wr.puts "#{i} And some more" >     } >     ruby\_to\_a\_wr.close > end > > Thread.new do >     while line = a\_err\_rd.gets >         puts "A err: #{line}" >     end >     a\_err\_rd.close > end > > Thread.new do >     while line = b\_err\_rd.gets >         puts "B err: #{line}" >     end >     b\_err\_rd.close > end > > Thread.new do >     while line = c\_err\_rd.gets >         puts "C err: #{line}" >     end >     c\_err\_rd.close > end > > while line = c\_to\_ruby\_rd.gets >     puts line > end > c\_to\_ruby\_rd.close ---
**Author:** ![Michal\_hramrach\_Such](https://avatars.discourse-cdn.com/v4/letter/m/3ab097/32.png) [@Michal\_hramrach\_Such](https://rubytalk.org/u/Michal_hramrach_Such)\ **Post date:** [12 May 2011 20:15 UTC](https://rubytalk.org/t/shell-pipeline-in-ruby/62696/20 "2011-05-12T20:15:12Z")
It provides stderr but not the pipeline which cannot be constructed in pure ruby (unless a popen taking fd:s for stdio is implemented). Thanks Michal > **···** > > On 12 May 2011 21:54, Stu \ wrote: > > > There is also popen4 which I believe also gives you the pid with std - > > in/out/err [Next page](https://rubytalk.org/t/shell-pipeline-in-ruby/62696.md?page=2)