Working with a slow pipe (IO.popen)

ruby-talk
1

I'm doing some work with pymol and having trouble getting all the
pymol output out:

# I open it in quiet/commandline mode with a pipe:
IO.popen("pymol -cq -p", 'w+') do |pipe|
  pipe.puts "load file.pdb, mymodel\n"
  pipe.puts "run my_script.py"
  # this command will generate a whole bunch of output to stdout
  pipe.puts "my_script mymodel"
  sleep(5) # <--- this is what I have to do in order to get the
output
  pipe.close_write
  output = pipe.read
end

What is annoying is that unless I sleep for 2-5 seconds I don't get
output or (even worse) my output is cut off. There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I've tried things like piping in "quit", but that doesn't seem to
work. Anybody know how to solve this? I'm happy to use a different
approach if it means I can be sure I get all of my output out. --
Thanks!

2

Hi,

I don't know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write. pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

  IO.popen("pymol -cq -p", 'w+') do |pipe|
    pipe.puts "load file.pdb, mymodel\n"
    pipe.puts "run my_script.py"
    pipe.puts "my_script mymodel"

    # IO.select([pipe]) # you shouldn't need to uncomment this...

    output = pipe.read
  end

···

bwv549 <jtprince@gmail.com> wrote:

I'm doing some work with pymol and having trouble getting all the
pymol output out:

# I open it in quiet/commandline mode with a pipe:
IO.popen("pymol -cq -p", 'w+') do |pipe|
  pipe.puts "load file.pdb, mymodel\n"
  pipe.puts "run my_script.py"
  # this command will generate a whole bunch of output to stdout
  pipe.puts "my_script mymodel"
  sleep(5) # <--- this is what I have to do in order to get the
output
  pipe.close_write
  output = pipe.read
end

What is annoying is that unless I sleep for 2-5 seconds I don't get
output or (even worse) my output is cut off. There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I've tried things like piping in "quit", but that doesn't seem to
work. Anybody know how to solve this? I'm happy to use a different
approach if it means I can be sure I get all of my output out. --
Thanks!

--
Eric Wong

3

both of those hang indefinitely. It doesn't hang if I add the line:

   pipe.puts "quit"

but that doesn't give me all the output either. So maybe pymol
behaves differently than many commandline programs?

Here is what I've worked out for the time being. It opens a thread
for reading the output and asks if there is any output every 1/2
second. Once we go 1/2 second without output we kill the thread.

my_string = ""
Open3.popen3("pymol -cq -p") do |si, so, se|
  si.puts "load file.pdb, mymodel\n"
  si.puts "run my_script.py\n"
  si.puts "myscript mymodel\n"

  forstdout = Thread.new do
    Thread.current['lines'] =
    while line = so.gets
      Thread.current['lines'] << line
    end
  end

  past_size = -1
  loop do
    sleep(0.5)
    current_size = forstdout['lines'].size
    break if current_size == past_size
    past_size = current_size
  end
  my_string = forstdout['lines']
  forstdout.kill
end

This is not the fastest method, but it does seem to work OK. I would
still love to hear other ideas.

···

On Dec 15, 11:32 pm, Eric Wong <normalper...@yhbt.net> wrote:

bwv549 <jtpri...@gmail.com> wrote:
> I'm doing some work with pymol and having trouble getting all the
> pymol output out:

> # I open it in quiet/commandline mode with a pipe:
> IO.popen("pymol -cq -p", 'w+') do |pipe|
> pipe.puts "load file.pdb, mymodel\n"
> pipe.puts "run my_script.py"
> # this command will generate a whole bunch of output to stdout
> pipe.puts "my_script mymodel"
> sleep(5) # <--- this is what I have to do in order to get the
> output
> pipe.close_write
> output = pipe.read
> end

> What is annoying is that unless I sleep for 2-5 seconds I don't get
> output or (even worse) my output is cut off. There has to be a way
> besides arbitrarily sleeping to ensure that the command is finished.
> I've tried things like piping in "quit", but that doesn't seem to
> work. Anybody know how to solve this? I'm happy to use a different
> approach if it means I can be sure I get all of my output out. --
> Thanks!

Hi,

I don't know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write. pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

IO.popen("pymol -cq -p", 'w+') do |pipe|
pipe.puts "load file.pdb, mymodel\n"
pipe.puts "run my_script.py"
pipe.puts "my_script mymodel"

\# IO\.select\(\[pipe\]\) \# you shouldn&#39;t need to uncomment this\.\.\.

output = pipe\.read

end

--
Eric Wong

4

> > I'm doing some work with pymol and having trouble getting all the
> > pymol output out:
>
> > # I open it in quiet/commandline mode with a pipe:
> > IO.popen("pymol -cq -p", 'w+') do |pipe|
> > pipe.puts "load file.pdb, mymodel\n"
> > pipe.puts "run my_script.py"
> > # this command will generate a whole bunch of output to stdout
> > pipe.puts "my_script mymodel"
> > sleep(5) # <--- this is what I have to do in order to get the
> > output
> > pipe.close_write
> > output = pipe.read
> > end
>
> > What is annoying is that unless I sleep for 2-5 seconds I don't get
> > output or (even worse) my output is cut off. There has to be a way
> > besides arbitrarily sleeping to ensure that the command is finished.
> > I've tried things like piping in "quit", but that doesn't seem to
> > work. Anybody know how to solve this? I'm happy to use a different
> > approach if it means I can be sure I get all of my output out. --
> > Thanks!
>
> Hi,
>
> I don't know about pymol, but some apps do not respond well to having
> its stdin closed on it with close_write. pipe.read should just block
> until you get output, otherwise put an IO.select([pipe) in front of it.
>
> Does this work?
>
> IO.popen("pymol -cq -p", 'w+') do |pipe|
> pipe.puts "load file.pdb, mymodel\n"
> pipe.puts "run my_script.py"
> pipe.puts "my_script mymodel"
>
> # IO.select([pipe]) # you shouldn't need to uncomment this...
>
> output = pipe.read
> end

both of those hang indefinitely. It doesn't hang if I add the line:

   pipe.puts "quit"

but that doesn't give me all the output either. So maybe pymol
behaves differently than many commandline programs?

Even IO.select([pipe]) hangs indefinitely? Oh, I guess pymol
expects an explicit quit of some sort.... What if you did:

  IO.select([pipe])
  pipe.readpartial(whatever_size_you_are_comfortable_with)

Here is what I've worked out for the time being. It opens a thread
for reading the output and asks if there is any output every 1/2
second. Once we go 1/2 second without output we kill the thread.

Here's a version without threads, should work the same:

  my_string = ""
  Open3.popen3("pymol -cq -p") do |si, so, se|
    si.puts "load file.pdb, mymodel\n"
    si.puts "run my_script.py\n"
    si.puts "myscript mymodel\n"

    # await input for 0.5 seconds, will return nil and
    # break the loop if there is nothing to read from so after 0.5s
    while ready = IO.select([so], nil, nil, 0.5)
      # ready.first == so # in this case

      # read until the current pipe buffer is empty
      begin
        my_string << so.read_nonblock(4096)
      rescue Errno::EAGAIN
        break
      end while true
    end
  end

···

bwv549 <jtprince@gmail.com> wrote:

On Dec 15, 11:32 pm, Eric Wong <normalper...@yhbt.net> wrote:
> bwv549 <jtpri...@gmail.com> wrote:

--
Eric Wong

5

> I'm doing some work with pymol and having trouble getting all the
> pymol output out:

> # I open it in quiet/commandline mode with a pipe:
> IO.popen("pymol -cq -p", 'w+') do |pipe|
> pipe.puts "load file.pdb, mymodel\n"
> pipe.puts "run my_script.py"
> # this command will generate a whole bunch of output to stdout
> pipe.puts "my_script mymodel"
> sleep(5) # <--- this is what I have to do in order to get the
> output
> pipe.close_write
> output = pipe.read
> end

> What is annoying is that unless I sleep for 2-5 seconds I don't get
> output or (even worse) my output is cut off. There has to be a way
> besides arbitrarily sleeping to ensure that the command is finished.
> I've tried things like piping in "quit", but that doesn't seem to
> work. Anybody know how to solve this? I'm happy to use a different
> approach if it means I can be sure I get all of my output out. --
> Thanks!

Hi,

I don't know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write. pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

IO.popen("pymol -cq -p", 'w+') do |pipe|
pipe.puts "load file.pdb, mymodel\n"
pipe.puts "run my_script.py"
pipe.puts "my_script mymodel"

\# IO\.select\(\[pipe\]\) \# you shouldn&#39;t need to uncomment this\.\.\.

output = pipe\.read

end

--
Eric Wong

both of those hang indefinitely. It doesn't hang if I add the line:

pipe.puts "quit"

but that doesn't give me all the output either. So maybe pymol
behaves differently than many commandline programs?

What does the documentation of "pymol" say?

Here is what I've worked out for the time being. It opens a thread
for reading the output and asks if there is any output every 1/2
second. Once we go 1/2 second without output we kill the thread.

my_string = ""
Open3.popen3("pymol -cq -p") do |si, so, se|
si.puts "load file.pdb, mymodel\n"
si.puts "run my_script.py\n"
si.puts "myscript mymodel\n"

forstdout = Thread.new do
Thread.current['lines'] =
while line = so.gets
Thread.current['lines'] << line
end
end

You don't need the thread local for this to work. Note there is also
Thread#value!

How does this work:

output = IO.popen("pymol -cq -p", 'w+') do |pipe|
  reader = Thread.new { pipe.to_a }

  pipe.puts "load file.pdb, mymodel"
  pipe.puts "run my_script.py"
  # this command will generate a whole bunch of output to stdout
  pipe.puts "my_script mymodel"

  pipe.close_write
  reader.value
end

Kind regards

robert

···

2009/12/16 bwv549 <jtprince@gmail.com>:

On Dec 15, 11:32 pm, Eric Wong <normalper...@yhbt.net> wrote:

bwv549 <jtpri...@gmail.com> wrote:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/

6

@Robert: I've looked through a lot of pymol docs and haven't found
anything on it yet, will post if I do.

The solution you post looks very intriguing. It runs fine, but closes
before I get back the output I need. It seems like there must be some
kind of a pause to let pymol start spitting out output. I may play
around with it some to see if I can get it working.

@Eric: this solution works nicely and returns all the expected
output. I will be using it unless something better comes along.

Thanks to both of you for your excellent input on this. --John

···

On Dec 16, 1:05 am, Robert Klemme <shortcut...@googlemail.com> wrote:

2009/12/16 bwv549 <jtpri...@gmail.com>:

> On Dec 15, 11:32 pm, Eric Wong <normalper...@yhbt.net> wrote:
>> bwv549 <jtpri...@gmail.com> wrote:
>> > I'm doing some work with pymol and having trouble getting all the
>> > pymol output out:

>> > # I open it in quiet/commandline mode with a pipe:
>> > IO.popen("pymol -cq -p", 'w+') do |pipe|
>> > pipe.puts "load file.pdb, mymodel\n"
>> > pipe.puts "run my_script.py"
>> > # this command will generate a whole bunch of output to stdout
>> > pipe.puts "my_script mymodel"
>> > sleep(5) # <--- this is what I have to do in order to get the
>> > output
>> > pipe.close_write
>> > output = pipe.read
>> > end

>> > What is annoying is that unless I sleep for 2-5 seconds I don't get
>> > output or (even worse) my output is cut off. There has to be a way
>> > besides arbitrarily sleeping to ensure that the command is finished.
>> > I've tried things like piping in "quit", but that doesn't seem to
>> > work. Anybody know how to solve this? I'm happy to use a different
>> > approach if it means I can be sure I get all of my output out. --
>> > Thanks!

>> Hi,

>> I don't know about pymol, but some apps do not respond well to having
>> its stdin closed on it with close_write. pipe.read should just block
>> until you get output, otherwise put an IO.select([pipe) in front of it.

>> Does this work?

>> IO.popen("pymol -cq -p", 'w+') do |pipe|
>> pipe.puts "load file.pdb, mymodel\n"
>> pipe.puts "run my_script.py"
>> pipe.puts "my_script mymodel"

>> # IO.select([pipe]) # you shouldn't need to uncomment this...

>> output = pipe.read
>> end

>> --
>> Eric Wong

> both of those hang indefinitely. It doesn't hang if I add the line:

> pipe.puts "quit"

> but that doesn't give me all the output either. So maybe pymol
> behaves differently than many commandline programs?

What does the documentation of "pymol" say?

> Here is what I've worked out for the time being. It opens a thread
> for reading the output and asks if there is any output every 1/2
> second. Once we go 1/2 second without output we kill the thread.

> my_string = ""
> Open3.popen3("pymol -cq -p") do |si, so, se|
> si.puts "load file.pdb, mymodel\n"
> si.puts "run my_script.py\n"
> si.puts "myscript mymodel\n"

> forstdout = Thread.new do
> Thread.current['lines'] =
> while line = so.gets
> Thread.current['lines'] << line
> end
> end

You don't need the thread local for this to work. Note there is also
Thread#value!

How does this work:

output = IO.popen("pymol -cq -p", 'w+') do |pipe|
reader = Thread.new { pipe.to_a }

pipe.puts "load file.pdb, mymodel"
pipe.puts "run my_script.py"
# this command will generate a whole bunch of output to stdout
pipe.puts "my_script mymodel"

pipe.close_write
reader.value
end

Kind regards

robert

--
remember.guy do |as, often| as.you_can - without endhttp://blog.rubybestpractices.com/

7

@Robert: I've looked through a lot of pymol docs and haven't found
anything on it yet, will post if I do.

The solution you post looks very intriguing. It runs fine, but closes
before I get back the output I need.

Frankly, I don't see how this can be: the reader thread should read *everthing* that comes out of the pipe and only stops if there is EOF. The only explanation I have so far is that maybe some CTRL-D or other EOF character is sent through the pipe making the reader think it has reached EOF.

It seems like there must be some
kind of a pause to let pymol start spitting out output. I may play
around with it some to see if I can get it working.

Even if there is a pause the reader must wait. There must be something else happening. Maybe try to do it on the shell and use 'od' to see what is sent across the pipe.

@Eric: this solution works nicely and returns all the expected
output. I will be using it unless something better comes along.

Good that you have a working solution. However, what you state worries me a bit. I believe we might not have identified the root cause of the issue.

Thanks to both of you for your excellent input on this. --John

You're welcome!

Kind regards

  robert

···

On 17.12.2009 06:40, bwv549 wrote:

--
remember.guy do |as, often| as.you_can - without end
http://blog.rubybestpractices.com/