Piping binary data to an external program

ruby-talk
1

Hello,

I'm working on a logparser and i've run into some issues. It will
parse OpenBSD PF logs. They are tcpdump format logs and BSD normally
compress them.

Here is the usage I have in mind:

"gzip -cd log.gz | ruby logparser.rb --today"

I have the following code:

Open3.popen3("/usr/sbin/tcpdump -nettr -") { |in_io, out_io, err_io|
  in_io.write($stdin.read)
  in_io.close
  $log = out_io.read
}

The script freezes on the open3 line and doesn't continue. I've tested
several other methods but it doesn't seem to work.

Any suggestions on how this can be done?

Regards,

Ricardo.

2

MKSM> wrote:
Hello,

I'm working on a logparser and i've run into some issues. It will
parse OpenBSD PF logs. They are tcpdump format logs and BSD normally
compress them.

Here is the usage I have in mind:

"gzip -cd log.gz | ruby logparser.rb --today"

I have the following code:

Open3.popen3("/usr/sbin/tcpdump -nettr -") { |in_io, out_io, err_io|
in_io.write($stdin.read)
in_io.close
$log = out_io.read
}

The script freezes on the open3 line and doesn't continue. I've tested
several other methods but it doesn't seem to work.

Any suggestions on how this can be done?

It's likely that you run into a deadlock caused by pipe buffer sizes. I
suggest to not read and write the whole content but to do it in chunks.
Also, I'd separate the reading and writing code into two threads.

Untested:

Open3.popen3("/usr/sbin/tcpdump -nettr -") { |in_io, out_io, err_io|
t = Thread.new(in_io) do |out|
  while ( buff = $stdin.read( 1024 ) )
    out.write(buff)
  end
  out.close
end
$log = ""
while ( b = out_io.read(1024))
  $log << b
end
}

HTH

Kind regards

    robert