Capture output

[Simon_Strandgaard]

I have seen much talking about this topic, but no working code!

I want to capture all output from [ruby, child_processes].
How can I do this ?

When I run my code I get this output…
But this is not what I am expecting.

ruby main.rb
stderr
stdout
defout
OK 1
OK 2

I want output like this… what am I doing wrong ?
OK 1
OK stderr
OK stdout
OK defout
OK 2

cat main.rb
require ‘stringio’

def capture
raise unless block_given?
a, b, c = $defout, $stderr, $stdout
buf = StringIO.new(s = “”, “w”)
begin
$defout, $stderr, $stdout = buf
yield
ensure
buf.close
$defout, $stderr, $stdout = a, b, c
end
s.split(/\n/)
end

res = capture {
puts “1”
system(“ruby output.rb”) # BOOM - Output is not being captured?
puts “2”
}
puts res.collect{|s| “OK “+s }.join(”\n”)

cat output.rb
$stderr.puts “stderr”
$stdout.puts “stdout”
$defout.puts “defout”

···

–
Simon Strandgaard

[Ara.T.Howard1 (Ara.T.Howard)]

$stdout and friends are PROCESS based, Kernel.system starts a new process with
it’s OWN $stdout and friends - they are not inherited. you could do something
like this:

~/eg/ruby > cat main.rb
require ‘stringio’

def capture
raise unless block_given?
a, b, c = $defout, $stderr, $stdout
buf = StringIO.new(s = “”, “w”)
begin
#$defout, $stderr, $stdout = buf
$defout, $stderr, $stdout = buf, buf, buf
yield
ensure
buf.close
$defout, $stderr, $stdout = a, b, c
end
s.split(/\n/)
end

res = capture {
puts “1”
#system(“ruby output.rb”) # BOOM - Output is not being captured?
load ‘output.rb’
puts “2”
}
puts res.collect{|s| “OK “+s }.join(”\n”)

~/eg/ruby > cat output.rb
$stderr.puts “stderr”
$stdout.puts “stdout”
$defout.puts “defout”

~/eg/ruby > ruby main.rb
OK 1
OK stderr
OK stdout
OK defout
OK 2

-a

···

On Wed, 7 May 2003, Simon Strandgaard wrote:

–

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
====================================

[Florian_G_Pflug]

I have seen much talking about this topic, but no working code!

I want to capture all output from [ruby, child_processes].
How can I do this ?

def capture
raise unless block_given?
a, b, c = $defout, $stderr, $stdout
buf = StringIO.new(s = “”, “w”)
begin
$defout, $stderr, $stdout = buf
You need to use $stdout.reopen(buf) here. Same for $stderr.

    yield
ensure
            buf.close
    $defout, $stderr, $stdout = a, b, c
end
    s.split(/\n/)

end

res = capture {
puts “1”
system(“ruby output.rb”) # BOOM - Output is not being captured?
puts “2”
}
puts res.collect{|s| “OK “+s }.join(”\n”)

cat output.rb
$stderr.puts “stderr”
$stdout.puts “stdout”
$defout.puts “defout”

But since IO::reopen seems to be a quite special function (it actually
replaces its receiver - it’s the only ruby function I know of that does
this), it might be that the argument has to be of a special class (maybe
derived from IO?).

I also never used stringio, so I can’t tell wheather it’s “compatible
enough” with IO to use it as a parameter to reopen().

Maybe someone with more knowledge of those ruby internals can clarify these
issues.

greetings, Florian Pflug

···

On Wed, May 07, 2003 at 07:53:05AM +0900, Simon Strandgaard wrote:

[Robert]

How about:

def capture
raise unless block_given?
a, b, c = $defout, $stderr, $stdout
buf = StringIO.new(s = “”, “w”)

begin
    $defout, $stderr, $stdout = buf
    yield
ensure
    buf.close
    $defout, $stderr, $stdout = a, b, c
end
s.split(/\n/)

end

res = capture {
puts “1”
popen(“ruby output.rb”, “r”) do |io|
print( io.read )
end
puts “2”
}

Regards

robert

“Simon Strandgaard” 0bz63fz3m1qt3001@sneakemail.com schrieb im
Newsbeitrag news:pan.2003.05.06.22.54.18.656367@sneakemail.com…

···

I have seen much talking about this topic, but no working code!

I want to capture all output from [ruby, child_processes].
How can I do this ?

When I run my code I get this output…
But this is not what I am expecting.

ruby main.rb
stderr
stdout
defout
OK 1
OK 2

I want output like this… what am I doing wrong ?
OK 1
OK stderr
OK stdout
OK defout
OK 2

cat main.rb
require ‘stringio’

def capture
raise unless block_given?
a, b, c = $defout, $stderr, $stdout
buf = StringIO.new(s = “”, “w”)
begin
$defout, $stderr, $stdout = buf
yield
ensure
buf.close
$defout, $stderr, $stdout = a, b, c
end
s.split(/\n/)
end

res = capture {
puts “1”
system(“ruby output.rb”) # BOOM - Output is not being captured?
puts “2”
}
puts res.collect{|s| “OK “+s }.join(”\n”)

cat output.rb
$stderr.puts “stderr”
$stdout.puts “stdout”
$defout.puts “defout”

–
Simon Strandgaard

[Simon_Strandgaard]

res = capture {
puts “1”
#system(“ruby output.rb”) # BOOM - Output is not being captured?
load ‘output.rb’
^^^^
^^^^

Sorry. Load does not solve it.
I want to execute arbitrary unix-programs and capture all their output.
This was just the simples way I could get different kinds of output.

    puts "2"

}
puts res.collect{|s| “OK “+s }.join(”\n”)

I am embedding ruby into another application and therefore I want
to redirect all rubys output elsewhere… so that ruby is 100% silent.

Things I know of which can generate output:
system(“echo hello”)
echo hello
puts
print
printf

How do I overload “system” and “backquote” ?

How do I ensure ruby stays 100% silent ???

···

On Wed, 07 May 2003 00:02:09 +0000, ahoward wrote:

–
Simon Strandgaard

[Simon_Strandgaard]

I want to capture all output from [ruby, child_processes].
How can I do this ?

You need to use $stdout.reopen(buf) here. Same for $stderr.

But since IO::reopen seems to be a quite special function (it actually
replaces its receiver - it’s the only ruby function I know of that does
this), it might be that the argument has to be of a special class (maybe
derived from IO?).

Yes…

The problem is almost solved - I found this (thanks paul brannan):
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&threadm=20020807095619.D4512%40atdesk.com&rnum=12&prev=/groups%3Fq%3Druby%2Bstringio%26hl%3Den%26lr%3D%26ie%3DUTF-8%26start%3D10%26sa%3DN

There seems to be some glitches where it does’nt capture all output.
I get 2 different kinds of output… here is the first output:

ruby main.rb
OK 1
OK stderrstdout
OK defout
OK
OK 2
OK stderr
OK stdout
OK defout
OK 3

The line “OK stderrstdout” is buggy…

The second output:

ruby main.rb
OK 1
OK stderr
OK stdout
OK defout
OK 2
OK stderr
OK stdout
OK defout
OK 3

Running the code several times, then you will see
that it varies between first&second output… Why ???

cat main.rb
require ‘stringio’

class Redirector
def initialize(from, to)
tmp = from.dup
r, w = IO.pipe
from.reopen(w)
@t = Thread.new do
begin
loop do
s = r.read(1) # TODO: can I make this buffered?
to.write(s)
end
ensure
from.reopen(tmp)
end
end
end
def stop
@t.kill
end
def self.redirect(from, to)
s = self.new(from, to)
begin
yield
ensure
s.stop
end
end
end

def capture
raise unless block_given?
a, b, c = $defout, $stderr, $stdout
buf = StringIO.new(s = “”, “w”)
rdr_serr = Redirector.new($stderr, buf)
rdr_sout = Redirector.new($stdout, buf)
begin
$defout = buf
yield
ensure
$defout, $stderr, $stdout = a, b, c
rdr_serr.stop
rdr_sout.stop
buf.close
end
s.split(/\n/)
end

res = capture {
puts “1”
system(“ruby output.rb”)
puts “2”
puts ruby output.rb
puts “3”
}
puts res.collect{|s| “OK “+s }.join(”\n”)

cat output.rb
$stderr.puts “stderr”
$stdout.puts “stdout”
$defout.puts “defout”

The remaining question: Why does the output differs when
I run the program several times ???

···

On Wed, 07 May 2003 09:33:31 +0900, Florian G. Pflug wrote:

On Wed, May 07, 2003 at 07:53:05AM +0900, Simon Strandgaard wrote:

–
Simon Strandgaard

[Simon_Strandgaard]

res = capture {
puts “1”

    popen("ruby output.rb", "r") do |io|
      print( io.read )
    end

      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I want to capture all output from ruby, so that
ruby cannot interfere with that application that
im embedding ruby into. Silencing ruby 100%!

This is not a solution to the problem
the output from
system(“true”)
true
will not get captured properly.

    puts "2"

}

Thanks for trying

···

On Wed, 07 May 2003 10:32:29 +0200, Robert Klemme wrote:

–
Simon Strandgaard

[Brian_Candler]

Because you are killing threads before they have finished doing their useful
work? The threads should exit themselves on seeing an EOF condition on the
read stream, then you can use t.join to wait for them.

But I wonder if you are over-complicating things, if you just want to
capture output into a string. How about:

mystring = command 2>&1

Otherwise, the standard way of doing this is with IO.popen: see
http://ruby-talk.org/68660

See the follow-ups for info on actually redirecting your stdout into a file.

Regards,

Brian.

···

On Wed, May 07, 2003 at 09:33:50AM +0900, Simon Strandgaard wrote:

The remaining question: Why does the output differs when
I run the program several times ???

[Brian_Candler]

$defout = $stdout = $stderr = File.open(“/dev/null”,“w”)

I don’t see the point though. If Ruby needs to report something on stderr,
then it’s probably worth seeing - just the same as if your C application
decided to print something on stderr.

Now, if you want to prevent the C app from outputing to stderr, then you’d
perhaps write something like

close(2);     /* close(stderr_fileno) is better */

or use dup2 to associate stderr with /dev/null, which should also do the
trick, since your embedded Ruby interpreter will share stderr with it.

This seems unrelated to capturing the output of child processes though;
that’s a different question, and is exactly the same as if you were spawning
a child process from C (using ‘system()’ for example). In other words, I
think it’s orthogonal to any Ruby issue.

Brian.

···

On Wed, May 07, 2003 at 05:14:57PM +0900, Simon Strandgaard wrote:

I want to capture all output from ruby, so that
ruby cannot interfere with that application that
im embedding ruby into. Silencing ruby 100%!

[Robert]

While experimenting a bit I discovered that this script hangs in the line
that puts “start” (version 1.6.7 on win and linux):

rd, wr = IO.pipe

reader = Thread.new do
buffer = “”

puts LINE

rd.each do |line|
puts LINE
buffer << line
end

buffer
end

last line printed:

puts LINE

wr.puts “start”

puts LINE

wr.flush
wr.close

puts LINE

puts “read: <<#{reader.value}>>”

puts LINE

Any ideas why?

robert

[Simon_Strandgaard]

The remaining question: Why does the output differs when
I run the program several times ???

Because you are killing threads before they have finished doing their useful
work? The threads should exit themselves on seeing an EOF condition on the
read stream, then you can use t.join to wait for them.

How do I detect EOF ?

But I wonder if you are over-complicating things, if you just want to
capture output into a string. How about:

mystring = command 2>&1

Not doable… The full story why I need to make ruby silent:
I am writing a tutorial on how to embed ruby into c++.
http://metaeditor.sf.net/embed/
I want the tutorial-readers to know that they don’t have to worry
about such “piping” issues and that ruby is behaving like an
c++ iostream, though which ALL output is delivered.
I assume that this “piping” stuff will scare most people from
attempting to embed ruby. Therefore this is not doable!

Otherwise, the standard way of doing this is with IO.popen: see
http://ruby-talk.org/68660
See the follow-ups for info on actually redirecting your stdout into a file.

As I said before the standard way is not doable!

I re-read the thread and found some good ideas

cat test.rb
$stdout = File.open(“log.txt”,“w”)
$stdout.sync = true # what is sync ???
$defout = $stderr = $stdout
puts “1”
system(“ls”)
puts “2”

This can capture everything, just what I need!
Now I just have to fit it into the earlier mentioned
code.

Could “sync” be the solution to the glitch problem ???

···

On Wed, 07 May 2003 17:44:22 +0900, Brian Candler wrote:

On Wed, May 07, 2003 at 09:33:50AM +0900, Simon Strandgaard wrote:

–
Simon Strandgaard

[Simon_Strandgaard]

I want to encapsulate ruby so its behaving like a
c++ iostream class… what I want is to redirect everything.

Thanks for your /dev/null efforts… but I cannot use this either

···

On Wed, 07 May 2003 20:12:16 +0900, Brian Candler wrote:

On Wed, May 07, 2003 at 05:14:57PM +0900, Simon Strandgaard wrote:

I want to capture all output from ruby, so that
ruby cannot interfere with that application that
im embedding ruby into. Silencing ruby 100%!

$defout = $stdout = $stderr = File.open(“/dev/null”,“w”)

–
Simon Strandgaard

[Carlos]

While experimenting a bit I discovered that this script hangs in the line
that puts “start” (version 1.6.7 on win and linux):
[…]
Any ideas why?

Doesn’t hang here…

$ ruby pipe.rb
6
17
21
9
26
read: <<start

30

$ ruby -v
ruby 1.6.8 (2002-12-24) [i386-linux]

Maybe you need to improve your ruby by 0.0.1

[Simon_Strandgaard]

Capturing output to a File works fine… But not to StringIO, Why ???

using File works as a charm:

cat short3.rb
$stdout = File.open(“log.txt”,“w”)

$defout = $stderr = $stdout
$stdout.sync = true
$stderr.sync = true
$defout.sync = true
puts “1”
system(“ruby output.rb”)
puts “2”
puts ruby output.rb
puts “3”

cat output.rb
$stderr.puts “stderr”
$stdout.puts “stdout”
$defout.puts “defout”
ruby short3.rb
cat log.txt
1
stderr
stdout
defout
2
stderr
stdout
defout
3

using StringIO causes trouble!
The content of “log.txt” is different… Why ???

cat short2.rb
require ‘stringio’
s = "test: "
$stdout = StringIO.open(s,“w”)

$defout = $stderr = $stdout
$stdout.sync = true
$stderr.sync = true
$defout.sync = true
puts “1”
system(“ruby output.rb”)
puts “2”
puts ruby output.rb
puts “3”

output result to file

f = File.open(“log.txt”, “w”)
f.puts s
f.close

cat output.rb
$stderr.puts “stderr”
$stdout.puts “stdout”
$defout.puts “defout”
ruby short2.rb
stderr
stdout
defout
stderr
cat log.txt
1
2
stdout
defout
3

What should I do in order to capture output to a string ???

···

–
Simon Strandgaard

[Brian_Candler]

http://www.rubycentral.com/book/ref_c_io.html

···

On Wed, May 07, 2003 at 05:35:14PM +0900, Simon Strandgaard wrote:

How do I detect EOF ?

[ts1 (ts)]

Doesn't hang here...

same doesn't hang

Maybe you need to improve your ruby by 0.0.1

no, no, work fine with 1.6.7

Guy Decoux

[Nobuyoshi_Nakada]

Hi,

···

At Wed, 7 May 2003 18:55:54 +0900, Simon Strandgaard wrote:

Capturing output to a File works fine… But not to StringIO, Why ???

StringIO works only in a process, not across processes.

–
Nobu Nakada

[Brian_Candler]

You can’t, in the way that you showed. You must use IO.popen as I said
before.

The problem is this: you are forking a child process, which inherits
stdin/stdout/stderr. It then writes to them using Unix write() calls, to
file descriptors 1 (stdout) and 2 (stderr).

Now, you can redirect fds 1/2 to a file before forking, in which case the
child will write to that file. But you cannot redirect those fds to a Ruby
object, because the Unix write() syscall does not call Ruby!

Now, if you use IO.popen, then you will set up an unnamed Unix pipe between
your application and the child. Any data written by the child will be sent
using write() into the pipe, and your application can pick it up and do
whatever it likes - such as appending it to a Ruby string.

However that’s only a single pipe, so that app that you spawn will still
have to remember to redirect stderr to stdout if you want to capture both.
The shell can do that for you using fd redirection:

data = nil
IO.popen(“cat /etc/shadow 2>&1”) do |f|
data = f.read
end
p data #>> “cat: /etc/shadow: Permission denied\n”

Note that if the command does not exist (e.g. replace ‘cat’ with ‘kat’) then
the shell will still write directly to its stderr.

But in general, I think it’s the right thing to let stderr do its job, of
giving an out-of-band error reporting channel.

Regards,

Brian.

···

On Wed, May 07, 2003 at 06:55:54PM +0900, Simon Strandgaard wrote:

What should I do in order to capture output to a string ???

[Robert]

A bit more testing reveals that this seems to be a problem of the mswin
version:

Microsoft Windows 2000 [Version 5.00.2195]
(C) Copyright 1985-2000 Microsoft Corp.

C:\WINNT>ruby --version
ruby 1.6.7 (2002-03-01) [i586-mswin32]

C:\WINNT>ruby c:\temp\x.rb
6
17
<>

Linux:

knoppix@ttyp0[knoppix]$ uname -a
Linux Knoppix 2.4.19-rc3-xfs #1 SMP Die Jul 23 21:13:33 CEST 2002 i686
unknown unknown GNU/Linux
knoppix@ttyp0[knoppix]$ ruby --version
ruby 1.6.7 (2002-03-19) [i386-linux]
knoppix@ttyp0[knoppix]$ ruby /mnt/hda1/temp/x.rb
6
17
21
9
26
read: <<start

30
knoppix@ttyp0[knoppix]$

I noticed also, that closing STDERR or $stderr does seem to close STDOUT,
too. Is this intended behavior or is it a known bug?

robert

“ts” decoux@moulon.inra.fr schrieb im Newsbeitrag
news:200305071636.h47GaiX26877@moulon.inra.fr…

···

Doesn’t hang here…

same doesn’t hang

Maybe you need to improve your ruby by 0.0.1

no, no, work fine with 1.6.7

Guy Decoux

[Simon_Strandgaard]

Do you have any suggestions for an alternative to StringIO ?

Instead of StringIO I tried inheriting from the IO class,
But no luck neither…

class CaptureOutput < IO
def initialize
super(2) # todo: how should I initialize the IO class ?
end
def write(text)
# redirect text elsewhere
end
end

How should I move on ?

···

On Wed, 07 May 2003 20:25:10 +0900, nobu.nokad wrote:

Hi,

At Wed, 7 May 2003 18:55:54 +0900, > Simon Strandgaard wrote:

Capturing output to a File works fine… But not to StringIO, Why ???

StringIO works only in a process, not across processes.

–
Simon Strandgaard