# Good library for command pipes in Ruby?
**URL:** https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306
**Category:** ruby-talk
**Created:** [17 October 2017 22:40 UTC](https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306 "2017-10-17T22:40:25Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Samuel\_Williams](https://rubytalk.org/u/Samuel_Williams)
#### Post date: [17 October 2017 22:40 UTC](https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306/1 "2017-10-17T22:40:25Z")
Hello,
I tried looking for a library that lets me write (lazy) command line pipes
in Ruby but came up short.
I would like to make a function that invokes a executable to transform an
object into text and either saves it to a file or dumps it out.
Ideally, I could write it like this:
def pipeline
return self.to\_text \>\> Process.spawn("dot")
end
Ideally, this pipeline is not invoked until the user does the following:
pipeline.write(path) //writes to file
or
pipeline.read //returns String
Does something like this exist already?
Thanks
Samuel
---
### Author:  [@Eric\_Wong](https://rubytalk.org/u/Eric_Wong)
#### Post date: [17 October 2017 23:06 UTC](https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306/2 "2017-10-17T23:06:06Z")
> Hello,
>
> I tried looking for a library that lets me write (lazy) command line pipes
> in Ruby but came up short.
>
> I would like to make a function that invokes a executable to transform an
> object into text and either saves it to a file or dumps it out.
Maybe using arrays to build up the command is enough;
at least that's all I use.
> Ideally, I could write it like this:
>
> def pipeline
> return self.to\_text \>\> Process.spawn("dot")
> end
>
> Ideally, this pipeline is not invoked until the user does the following:
>
> pipeline.write(path) //writes to file
command = # ... build the command array...
Then maybe something like:
IO.pipe do |r, w|
th = Thread.new { w.write(self.to\_text) }
pid = Process.spawn(env, \*command, in: r, out: path)
th.join
Process.waitpid2(pid)
end
[\*] of course, I'd like to get Thriber accepted to make the above
cheaper with s/Thread/Thriber/: [https://bugs.ruby-lang.org/issues/13618](https://bugs.ruby-lang.org/issues/13618)
Too late for 2.5, I guess :\<
> or
>
> pipeline.read //returns String
IO.popen(..., &:read) and `backtick` work well for me with tiny
command outputs; I prefer IO.popen with arrays to avoid
escaping.
But one of the best things about pipes (and IO objects in
general) is doing incremental processing for large outputs or
inputs. RAM is precious to me and I frequently work with
large files, so I prefer to use IO#readpartial or similar.
Unbound IO#read and #gets gives me nightmares :\<
> Does something like this exist already?
I started working along building complex pipelines as arrays
(and also throwing Procs anywhere in the pipeline) at:
[http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/435624](http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/435624)
And eventually for dtas \<[https://80x24.org/dtas/>:](https://80x24.org/dtas/>:)
[https://80x24.org/dtas-all/20170428200809.28822-1-e@80x24.org/raw](https://80x24.org/dtas-all/20170428200809.28822-1-e@80x24.org/raw)
But haven't really gotten around to doing more along those lines.
> **···**
>
> Samuel Williams \ wrote:
---
### Author:  [@Samuel\_Williams](https://rubytalk.org/u/Samuel_Williams)
#### Post date: [18 October 2017 01:16 UTC](https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306/3 "2017-10-18T01:16:54Z")
Interesting.. I started hacking something together, it's pretty small, but
it looks like this:
pipeline = Process::Pipeline.("cat Rakefile").("strings").("sort")
pipeline \> "output.txt"
Or...
buffer = pipeline.read
It composes together immutable steps, so you can compose pipes together,
e.g.
pipeline2 = pipeline.("tee foo.txt")
the original pipeline is shared but not modified in any way.
What do you think?
> **···**
>
> On 18 October 2017 at 12:06, Eric Wong \ wrote:
>
> > Samuel Williams \ wrote:
> > \> Hello,
> > \>
> > \> I tried looking for a library that lets me write (lazy) command line
> > pipes
> > \> in Ruby but came up short.
> > \>
> > \> I would like to make a function that invokes a executable to transform an
> > \> object into text and either saves it to a file or dumps it out.
> >
> > Maybe using arrays to build up the command is enough;
> > at least that's all I use.
> >
> > \> Ideally, I could write it like this:
> > \>
> > \> def pipeline
> > \> return self.to\_text \>\> Process.spawn("dot")
> > \> end
> > \>
> > \> Ideally, this pipeline is not invoked until the user does the following:
> > \>
> > \> pipeline.write(path) //writes to file
> >
> > command = # ... build the command array...
> >
> > Then maybe something like:
> >
> > IO.pipe do |r, w|
> > th = Thread.new { w.write(self.to\_text) }
> > pid = Process.spawn(env, \*command, in: r, out: path)
> > th.join
> > Process.waitpid2(pid)
> > end
> >
> > [\*] of course, I'd like to get Thriber accepted to make the above
> > cheaper with s/Thread/Thriber/: [https://bugs.ruby-lang.org/](https://bugs.ruby-lang.org/)
> > issues/13618
> > Too late for 2.5, I guess :\<
> >
> > \> or
> > \>
> > \> pipeline.read //returns String
> >
> > IO.popen(..., &:read) and `backtick` work well for me with tiny
> > command outputs; I prefer IO.popen with arrays to avoid
> > escaping.
> >
> > But one of the best things about pipes (and IO objects in
> > general) is doing incremental processing for large outputs or
> > inputs. RAM is precious to me and I frequently work with
> > large files, so I prefer to use IO#readpartial or similar.
> >
> > Unbound IO#read and #gets gives me nightmares :\<
> >
> > \> Does something like this exist already?
> >
> > I started working along building complex pipelines as arrays
> > (and also throwing Procs anywhere in the pipeline) at:
> >
> > [http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/435624](http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/435624)
> >
> > And eventually for dtas \<[https://80x24.org/dtas/>:](https://80x24.org/dtas/>:)
> >
> > [https://80x24.org/dtas-all/20170428200809.28822-1-e@80x24.org/raw](https://80x24.org/dtas-all/20170428200809.28822-1-e@80x24.org/raw)
> >
> > But haven't really gotten around to doing more along those lines.
> >
> > Unsubscribe: \
> > \<[http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>](http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>);
---
### Author:  [@Eric\_Wong](https://rubytalk.org/u/Eric_Wong)
#### Post date: [18 October 2017 02:21 UTC](https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306/4 "2017-10-18T02:21:23Z")
> Interesting.. I started hacking something together, it's pretty small, but
> it looks like this:
>
> pipeline = Process::Pipeline.("cat Rakefile").("strings").("sort")
> pipeline \> "output.txt"
Maybe only support arrays (instead of string commands) to avoid
shell injection problems. Ruby is especially great (compared to Perl5)
since we even have %W for expressing arrays with variables.
> Or...
>
> buffer = pipeline.read
Streaming support is most important:
pipeline.each\_chunk(size: 16384) { |buf| ... }
pipeline.each\_line(limit: 4096) { |line| ... }
> It composes together immutable steps, so you can compose pipes together,
> e.g.
>
> pipeline2 = pipeline.("tee foo.txt")
>
> the original pipeline is shared but not modified in any way.
>
> What do you think?
\*shrug\* I think I'd still rather be manipulating arrays, since
it's less new stuff to learn API-wise. And punctuation-oriented
APIs make searching harder, but maybe that's just me...
> **···**
>
> Samuel Williams \ wrote:
---
### Author:  [@Samuel\_Williams](https://rubytalk.org/u/Samuel_Williams)
#### Post date: [18 October 2017 02:40 UTC](https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306/5 "2017-10-18T02:40:10Z")
So, the commands are passed directly to `Process.spawn` so if you want to
write your command like ('cat', arg) it would work - the full args could be
([env,] \*command, \*\*options) off the top of my head.
It should be possible to implement each\_line and each\_chunk - I think these
are good ideas.
> **···**
>
> On 18 October 2017 at 15:21, Eric Wong \ wrote:
>
> > Samuel Williams \ wrote:
> > \> Interesting.. I started hacking something together, it's pretty small,
> > but
> > \> it looks like this:
> > \>
> > \> pipeline = Process::Pipeline.("cat Rakefile").("strings").("sort")
> > \> pipeline \> "output.txt"
> >
> > Maybe only support arrays (instead of string commands) to avoid
> > shell injection problems. Ruby is especially great (compared to Perl5)
> > since we even have %W for expressing arrays with variables.
> >
> > \> Or...
> > \>
> > \> buffer = pipeline.read
> >
> > Streaming support is most important:
> >
> > pipeline.each\_chunk(size: 16384) { |buf| ... }
> > pipeline.each\_line(limit: 4096) { |line| ... }
> >
> > \> It composes together immutable steps, so you can compose pipes together,
> > \> e.g.
> > \>
> > \> pipeline2 = pipeline.("tee foo.txt")
> > \>
> > \> the original pipeline is shared but not modified in any way.
> > \>
> > \> What do you think?
> >
> > \*shrug\* I think I'd still rather be manipulating arrays, since
> > it's less new stuff to learn API-wise. And punctuation-oriented
> > APIs make searching harder, but maybe that's just me...
> >
> > Unsubscribe: \
> > \<[http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>](http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>);
---
### Author:  [@Andy\_Jones](https://rubytalk.org/u/Andy_Jones)
#### Post date: [18 October 2017 07:42 UTC](https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306/6 "2017-10-18T07:42:41Z")
You might want to take a look at [GitHub - piotrmurach/tty-command: Execute shell commands with pretty output logging and capture stdout, stderr and exit status.](https://github.com/piotrmurach/tty-command) -- not sure if it does what you need, but it might be of some help.
> **···**
>
> From: ruby-talk [mailto:ruby-talk-bounces@ruby-lang.org] On Behalf Of Samuel Williams
> Sent: 18 October 2017 03:40
> To: Ruby users
> Subject: Re: Good library for command pipes in Ruby?
>
> So, the commands are passed directly to `Process.spawn` so if you want to write your command like ('cat', arg) it would work - the full args could be ([env,] \*command, \*\*options) off the top of my head.
>
> It should be possible to implement each\_line and each\_chunk - I think these are good ideas.
>
> On 18 October 2017 at 15:21, Eric Wong \\> wrote:
> Samuel Williams \\> wrote:
>
> > Interesting.. I started hacking something together, it's pretty small, but
> > it looks like this:
> >
> > pipeline = Process::Pipeline.("cat Rakefile").("strings").("sort")
> > pipeline \> "output.txt"
>
> Maybe only support arrays (instead of string commands) to avoid
> shell injection problems. Ruby is especially great (compared to Perl5)
> since we even have %W for expressing arrays with variables.
>
> > Or...
> >
> > buffer = pipeline.read
>
> Streaming support is most important:
>
> pipeline.each\_chunk(size: 16384) { |buf| ... }
> pipeline.each\_line(limit: 4096) { |line| ... }
>
> > It composes together immutable steps, so you can compose pipes together,
> > e.g.
> >
> > pipeline2 = pipeline.("tee foo.txt")
> >
> > the original pipeline is shared but not modified in any way.
> >
> > What do you think?
>
> \*shrug\* I think I'd still rather be manipulating arrays, since
> it's less new stuff to learn API-wise. And punctuation-oriented
> APIs make searching harder, but maybe that's just me...
>
> Unsubscribe: \?subject=unsubscribe\>
> \<[http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>](http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>);
>
> Click here to view Company Information and Confidentiality Notice.\<[http://www.jameshall.co.uk/index.php/small-print/email-disclaimer>](http://www.jameshall.co.uk/index.php/small-print/email-disclaimer>);
---
### Author:  [@Samuel\_Williams](https://rubytalk.org/u/Samuel_Williams)
#### Post date: [18 October 2017 08:47 UTC](https://rubytalk.org/t/good-library-for-command-pipes-in-ruby/74306/7 "2017-10-18T08:47:36Z")
Cool, I like the TTY series of gems, they are pretty useful for scripts..
Well, I did finally release something
[GitHub - ioquatix/process-pipeline](https://github.com/ioquatix/process-pipeline) it was just my side project
for the day, I think it's useful.
Thanks for everyone's ideas 🙂
> **···**
>
> On 18 October 2017 at 20:42, Andy Jones \ wrote:
>
> > You might want to take a look at [https://github.com/](https://github.com/)
> > piotrmurach/tty-command -- not sure if it does what you need, but it
> > might be of some help.
> >
> > \*From:\* ruby-talk [mailto:ruby-talk-bounces@ruby-lang.org] \*On Behalf Of \*Samuel
> > Williams
> > \*Sent:\* 18 October 2017 03:40
> > \*To:\* Ruby users
> > \*Subject:\* Re: Good library for command pipes in Ruby?
> >
> > So, the commands are passed directly to `Process.spawn` so if you want to
> > write your command like ('cat', arg) it would work - the full args could be
> > ([env,] \*command, \*\*options) off the top of my head.
> >
> > It should be possible to implement each\_line and each\_chunk - I think
> > these are good ideas.
> >
> > On 18 October 2017 at 15:21, Eric Wong \ wrote:
> >
> > Samuel Williams \ wrote:
> > \> Interesting.. I started hacking something together, it's pretty small,
> > but
> > \> it looks like this:
> > \>
> > \> pipeline = Process::Pipeline.("cat Rakefile").("strings").("sort")
> > \> pipeline \> "output.txt"
> >
> > Maybe only support arrays (instead of string commands) to avoid
> > shell injection problems. Ruby is especially great (compared to Perl5)
> > since we even have %W for expressing arrays with variables.
> >
> > \> Or...
> > \>
> > \> buffer = pipeline.read
> >
> > Streaming support is most important:
> >
> > pipeline.each\_chunk(size: 16384) { |buf| ... }
> > pipeline.each\_line(limit: 4096) { |line| ... }
> >
> > \> It composes together immutable steps, so you can compose pipes together,
> > \> e.g.
> > \>
> > \> pipeline2 = pipeline.("tee foo.txt")
> > \>
> > \> the original pipeline is shared but not modified in any way.
> > \>
> > \> What do you think?
> >
> > \*shrug\* I think I'd still rather be manipulating arrays, since
> > it's less new stuff to learn API-wise. And punctuation-oriented
> > APIs make searching harder, but maybe that's just me...
> >
> > Unsubscribe: \
> > \<[http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>](http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>);
> >
> > Click here to view Company Information and Confidentiality Notice.
> > \<[http://www.jameshall.co.uk/index.php/small-print/email-disclaimer>](http://www.jameshall.co.uk/index.php/small-print/email-disclaimer>);
> >
> > Unsubscribe: \
> > \<[http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>](http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-talk>);