Hi all,
I want to use Kernel#system or one of its friends to manipulate some data I have in a ruby string. Ideally, I'd want the executed command to read the string from stdin, as if ruby had piped the data to it. The data may be reasonably large. Any tips?
thanks
Caio Chassot wrote:
Hi all,
I want to use Kernel#system or one of its friends to manipulate some
data I have in a ruby string. Ideally, I'd want the executed command
to read the string from stdin, as if ruby had piped the data to it.
The data may be reasonably large. Any tips?
Could you be a bit more clear? Do you want to write a program that will
cooperate with a stream? As in:
data_source | ruby-program | data_dest
Yes?
--
Paul Lutus
http://www.arachnoid.com
http://www.ruby-doc.org/core/classes/IO.html#M001247
Hope this is what your after.
On 9/20/06, Caio Chassot <lists@v2studio.com> wrote:
Hi all,
I want to use Kernel#system or one of its friends to manipulate some
data I have in a ruby string. Ideally, I'd want the executed command
to read the string from stdin, as if ruby had piped the data to it.
The data may be reasonably large. Any tips?thanks
You could try using IO#popen The docs are at
No, from within a ruby program I want to call some shell command to act on data stored in a ruby string, so, in the ruby program, something like, in pseudoruby:
output = system('shell_cmd --params etc', :input => a_string)
On 2006-09-19, at 21:45 , Paul Lutus wrote:
Could you be a bit more clear? Do you want to write a program that will
cooperate with a stream? As in:data_source | ruby-program | data_dest
Yes?
Looks like it. Thanks a lot.
Do you know if it places the exit code in $? ?
I wonder what happens if the subprocess is killed by the system?
I guess I'll run some tests, but if you happen to know the answers, I'd be glad to hear them nonetheless.
On 2006-09-19, at 21:51 , Daniel N wrote:
You could try using IO#popen The docs are at
Hope this is what your after.
I haven't had a lot of experience with it personally. I came across it in a
plugin I was using that piped to an external progam and had a bit of a
look. The plugin doesn't seem to do anything special with it and I havn't
had a need to use it otherwise.
Sorri I can't help further.
On 9/20/06, Caio Chassot <lists@v2studio.com> wrote:
On 2006-09-19, at 21:51 , Daniel N wrote:
>>
>> You could try using IO#popen The docs are at
>
> class IO - RDoc Documentation
>
> Hope this is what your after.Looks like it. Thanks a lot.
Do you know if it places the exit code in $? ?
I wonder what happens if the subprocess is killed by the system?
I guess I'll run some tests, but if you happen to know the answers,
I'd be glad to hear them nonetheless.
Caio Chassot wrote:
Could you be a bit more clear? Do you want to write a program that
will
cooperate with a stream? As in:data_source | ruby-program | data_dest
Yes?
No, from within a ruby program I want to call some shell command to
act on data stored in a ruby string, so, in the ruby program,
something like, in pseudoruby:output = system('shell_cmd --params etc', :input => a_string)
Okay, try this:
On 2006-09-19, at 21:45 , Paul Lutus wrote:
--------------------------------
#!/usr/bin/ruby -w
s = "avast arlington always"
output = `echo #{s} | perl -pe "s/a/x/g"`
p output
--------------------------------
Output:
"xvxst xrlington xlwxys\n"
I want to emphasize there are a dozen ways to do this. This one happens to
be simple to understand. Another way is File.popen(), but that approach
makes bidirectional communications difficult.
--
Paul Lutus
http://www.arachnoid.com
Caio Chassot wrote:
You could try using IO#popen The docs are at
http://www.ruby-doc.org/core/classes/IO.html#M001247
Hope this is what your after.
Looks like it. Thanks a lot.
Do you know if it places the exit code in $? ?
I wonder what happens if the subprocess is killed by the system?
I guess I'll run some tests, but if you happen to know the answers,
I'd be glad to hear them nonetheless.
The problem with File.popen() is that you want bidirectional communication
with the child process, and AFAIK that doesn't offer it -- you can either
read, or write, to the pipe, but not both.
harp:~ > cat a.rb
require 'open4' # gem install open4
require 'yaml'
stdin = 'foobar'
stdout = ''
stderr = ''
Open4.spawn 'cat', :stdin => stdin, :stdout => stdout, :stderr => stderr
y 'stdout' => stdout
y 'stderr' => stderr
harp:~ > ruby a.rb
On Wed, 20 Sep 2006, Caio Chassot wrote:
On 2006-09-19, at 21:45 , Paul Lutus wrote:
Could you be a bit more clear? Do you want to write a program that will
cooperate with a stream? As in:data_source | ruby-program | data_dest
Yes?
No, from within a ruby program I want to call some shell command to act on data stored in a ruby string, so, in the ruby program, something like, in pseudoruby:
output = system('shell_cmd --params etc', :input => a_string)
---
stdout: foobar
---
stderr: ''
-a
--
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei
I thought about doing it with echo but then I'd have to escape the string, and what if I was dealing with binary data, and all that... so popen looks like a more robust solution.
On 2006-09-19, at 22:05 , Paul Lutus wrote:
output = `echo #{s} | perl -pe "s/a/x/g"`
I want to emphasize there are a dozen ways to do this. This one happens to
be simple to understand. Another way is File.popen(), but that approach
makes bidirectional communications difficult.
Hmm, that's not cool. I'll run some tests, but looks like it's back to using tempfiles then.
On 2006-09-19, at 22:10 , Paul Lutus wrote:
The problem with File.popen() is that you want bidirectional communication
with the child process, and AFAIK that doesn't offer it -- you can either
read, or write, to the pipe, but not both.
Again, I havn't used it before, but the IO#pipe command seems to allow for
bi-directional control.
http://www.ruby-doc.org/core/classes/IO.html#M001252
On 9/20/06, Caio Chassot <lists@v2studio.com> wrote:
On 2006-09-19, at 22:10 , Paul Lutus wrote:
> The problem with File.popen() is that you want bidirectional
> communication
> with the child process, and AFAIK that doesn't offer it -- you can
> either
> read, or write, to the pipe, but not both.Hmm, that's not cool. I'll run some tests, but looks like it's back
to using tempfiles then.
Caio Chassot wrote:
The problem with File.popen() is that you want bidirectional
communication
with the child process, and AFAIK that doesn't offer it -- you can
either
read, or write, to the pipe, but not both.Hmm, that's not cool. I'll run some tests, but looks like it's back
to using tempfiles then.
Did you read my other post? Piping information to child processes is easy.
bigstring = (lots of data)
output = `echo "#{bigstring}" | child_program_name`
This has every advantage over using temp files.
Caio Chassot wrote:
output = `echo #{s} | perl -pe "s/a/x/g"`
I want to emphasize there are a dozen ways to do this. This one
happens to
be simple to understand. Another way is File.popen(), but that
approach
makes bidirectional communications difficult.I thought about doing it with echo but then I'd have to escape the
string, and what if I was dealing with binary data, and all that...
so popen looks like a more robust solution.
Yes, except for the fact that you cannot easily find out what the child
process created as a result. But yes, if binary data is an issue,
File.popen() will reliably stream the data for you.
Yea, I figured out how to open a pipe for both read and write. You use a numeric mode. Here's what I got:
#!/usr/bin/env ruby
io = File.popen('perl -pe s/a/b/g', 0666)
io.puts('abc')
io.close_write
puts io.read
outputs:
bbc
So it's fine for my needs. I just need to wrap it into something more idiomatic.
Thanks Paul and Daniel
On 2006-09-19, at 22:24 , Daniel N wrote:
Again, I havn't used it before, but the IO#pipe command seems to allow for
bi-directional control.
you're making it pretty hard on yourself:
harp:~ > cat a.rb
cat = IO.popen 'cat', 'r+'
cat.sync = true
cat.puts 'foobar'
puts cat.gets
cat.puts 'barfoo'
puts cat.gets
harp:~ > ruby a.rb
foobar
barfoo
my session and open4 lib both abstract this largely, including a good deal of
error handling. there is also the built-in open3 module. check out the docs.
regards.
-a
On Wed, 20 Sep 2006, Caio Chassot wrote:
On 2006-09-19, at 22:24 , Daniel N wrote:
Again, I havn't used it before, but the IO#pipe command seems to allow for
bi-directional control.Yea, I figured out how to open a pipe for both read and write. You use a numeric mode. Here's what I got:
#!/usr/bin/env ruby
io = File.popen('perl -pe s/a/b/g', 0666)
io.puts('abc')
io.close_write
puts io.readoutputs:
bbcSo it's fine for my needs. I just need to wrap it into something more idiomatic.
Thanks Paul and Daniel
--
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei
you're making it pretty hard on yourself:
I guess I was 
my session and open4 lib both abstract this largely, including a good deal of
error handling. there is also the built-in open3 module. check out the docs.
Thanks.
Having a separate stdout and stderr is great, in fact, just the next thing I was going to ask about. So both open3 and open4 seem like great candidates.
Could you take some time to sell open4 over open3?
The main advantage I see is the greater abstraction, i.e. the spawn method.
On 2006-09-19, at 23:17 , ara.t.howard@noaa.gov wrote:
Interestingly, I just found a use for popen this morning. I was
playing around with letting open-uri use an 'scp' url.
This is a complete hack based on the URI::HTTP class, and it's pretty
rudimentary. Even though it's an 'scp' uri, it uses ssh internally.
As far as I can tell, without looking too hard, open-uri only supports
opening a URI for reading.
rick@frodo:/public/rubyscripts$ cat uris/scp.rb
On 9/19/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:
you're making it pretty hard on yourself:
harp:~ > cat a.rb
cat = IO.popen 'cat', 'r+'
#
# = uri/scp.rb
#
# Author:: Rick DeNatale
# License:: You can redistribute it and/or modify it under the same
term as Ruby.
#
require 'uri/generic'
module URI
# scp_URI = "scp://" [ userinfo "@" ] host [ ":" port ]
# [ ; parameter = value ] [ abs_path ]
class SCP < Generic
DEFAULT_PORT = 22
COMPONENT = [
:scheme,
:userinfo, :host, :port,
:path
].freeze
#
# == Description
#
# Create a new URI::SCP object from components of URI::SCP with
# check. It is scheme, userinfo, host, port, and path. It
provided by an Array of a Hash.
#
def self.build(args)
tmp = Util::make_components_hash(self, args)
return super(tmp)
end
#
# == Description
#
# Create a new URI::SCP object from ``generic'' components with no
# check.
#
def initialize(*arg)
super(*arg)
end
def ssh_command
result = ["ssh "]
result << "-p" + port.to_s + " " unless port == 22
result << user + "@" if user
result << host + " cat ~"
result << path
result.join('')
end
def open(&b)
IO.popen(ssh_command, &b)
end
end
@@schemes['SCP'] = SCP
end
--
Rick DeNatale
My blog on Ruby
http://talklikeaduck.denhaven2.com/
- you get the child pid returned also with open4
- a command that fails to exec raises and error (note this is sent back from
the child fork)
- when using the spawn method a great deal of abstraction is provided, not
least of which that stdin, stdout, stderr are each handled via a thread so
you may provide objects that can handle async output
Open4.spawn cmd, 1=>stdout_handler 2=>stderr_handler
and it will be passed to the hanlder in an async way
the main reason, however, is having the pid - it's hard to kill a proces
without it! 
that said - if all you require is stdout/stderr stay with the stdlib.
regards.
-a
On Wed, 20 Sep 2006, Caio Chassot wrote:
On 2006-09-19, at 23:17 , ara.t.howard@noaa.gov wrote:
you're making it pretty hard on yourself:
I guess I was
my session and open4 lib both abstract this largely, including a good deal of
error handling. there is also the built-in open3 module. check out the docs.Thanks.
Having a separate stdout and stderr is great, in fact, just the next thing I was going to ask about. So both open3 and open4 seem like great candidates.
Could you take some time to sell open4 over open3?
The main advantage I see is the greater abstraction, i.e. the spawn method.
--
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei
Is Open4 not win32 compatable? With all win32ole hooks, I'm suprised
no one has tackled this yet.
Ara, you helped me a while back with spawning processes in win32,
catching pids and error levels.
--just assumed if I could install it as a gem, it would work.
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
C:\Documents and Settings\d0t1q>gem install open4
Attempting local installation of 'open4'
Local gem file not found: open4*.gem
Attempting remote installation of 'open4'
Updating Gem source index for: http://gems.rubyforge.org
Successfully installed open4-0.7.0
C:\Documents and Settings\d0t1q>irb
irb(main):001:0> require 'open4'
=> false
irb(main):002:0> Open4.spawn "echo", :stdin => "hello"
NameError: uninitialized constant Fcntl::F_SETFD
from c:/ruby/lib/ruby/gems/1.8/gems/open4-0.7.0/lib/open4.rb:19:in `pope
n4'
from c:/ruby/lib/ruby/gems/1.8/gems/open4-0.7.0/lib/open4.rb:263:in `spa
wn'
from c:/ruby/lib/ruby/1.8/timeout.rb:48:in `timeout'
from c:/ruby/lib/ruby/gems/1.8/gems/open4-0.7.0/lib/open4.rb:262:in `spa
wn'
from (irb):2
irb(main):003:0>
On 9/20/06, ara.t.howard@noaa.gov <ara.t.howard@noaa.gov> wrote:
On Wed, 20 Sep 2006, Caio Chassot wrote:
>
> On 2006-09-19, at 23:17 , ara.t.howard@noaa.gov wrote:
>
>> you're making it pretty hard on yourself:
>
> I guess I was
>
>> my session and open4 lib both abstract this largely, including a good deal
>> of
>> error handling. there is also the built-in open3 module. check out the
>> docs.
>
> Thanks.
>
> Having a separate stdout and stderr is great, in fact, just the next thing I
> was going to ask about. So both open3 and open4 seem like great candidates.
>
> Could you take some time to sell open4 over open3?
>
> The main advantage I see is the greater abstraction, i.e. the spawn method.- you get the child pid returned also with open4
- a command that fails to exec raises and error (note this is sent back from
the child fork)- when using the spawn method a great deal of abstraction is provided, not
least of which that stdin, stdout, stderr are each handled via a thread so
you may provide objects that can handle async outputOpen4.spawn cmd, 1=>stdout_handler 2=>stderr_handler
and it will be passed to the hanlder in an async way
the main reason, however, is having the pid - it's hard to kill a proces
without it!that said - if all you require is stdout/stderr stay with the stdlib.
regards.
-a
--
in order to be effective truth must penetrate like an arrow - and that is
likely to hurt. -- wei wu wei