Zombie invasion - six ways to invoke shell processes

ruby-talk
1

I've got code somebody else wrote. The code uses exec() to invoke
shell processes. It spawns zombies by the thousands.

I need to redo this over the long term to stop using shell processes at all.

I need it to stop spawning zombies immediately.

This blog post gives six ways to invoke a shell process:

http://pasadenarb.com/2007/03/ruby-shell-commands.html

How do I choose which of these six methods to use?

All I remember off the top of my head is that backticks is the most
terse and system() is the one I use the most. I had forgotten exec()
even existed. "The Ruby Way" says that exec() doesn't use a subshell
and system() does. I think that's *why* I forgot exec() existed - I
think I blacklisted the method from the moment I discovered it.

I'm switching the legacy code to use system(). Way too much
superstition and guessing in this process for my taste though. How do
you choose the best (least zombie-prone) method to invoke a shell
process?

···

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org

2

It sounds like your use cases are rather simple and really only need to be focusing on `` and system().
In this case use `` when you need to capture the output and system() otherwise.

Of course I'm assuming that your need is simple here.

···

On Jul 05, 2007, at 15:24 , Giles Bowkett wrote:

I've got code somebody else wrote. The code uses exec() to invoke
shell processes. It spawns zombies by the thousands.

I need to redo this over the long term to stop using shell processes at all.

I need it to stop spawning zombies immediately.

This blog post gives six ways to invoke a shell process:

http://pasadenarb.com/2007/03/ruby-shell-commands.html

How do I choose which of these six methods to use?

All I remember off the top of my head is that backticks is the most
terse and system() is the one I use the most. I had forgotten exec()
even existed. "The Ruby Way" says that exec() doesn't use a subshell
and system() does. I think that's *why* I forgot exec() existed - I
think I blacklisted the method from the moment I discovered it.

I'm switching the legacy code to use system(). Way too much
superstition and guessing in this process for my taste though. How do
you choose the best (least zombie-prone) method to invoke a shell
process?

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org

--
Wayne E. Seguin
Sr. Systems Architect & Systems Admin
wayneseguin@gmail.com

3

I wonder if exec() might be useful in forking. Just playing around:

fork do
  exec 'ls'
end
puts "I can has ls?"
Process.wait

···

On 7/5/07, Giles Bowkett <gilesb@gmail.com> wrote:

All I remember off the top of my head is that backticks is the most
terse and system() is the one I use the most. I had forgotten exec()
even existed. "The Ruby Way" says that exec() doesn't use a subshell
and system() does. I think that's *why* I forgot exec() existed - I
think I blacklisted the method from the moment I discovered it.

4

Giles Bowkett wrote:

I've got code somebody else wrote. The code uses exec() to invoke
shell processes. It spawns zombies by the thousands.

I need to redo this over the long term to stop using shell processes at all.

why? it's one of ruby's strengths that it can act as glue between external components rather than having to code everything yourself.

I need it to stop spawning zombies immediately.

Look for the cause. Zombie processes are child processes whose return response has not yet been collected by the parent. Since you say that the code uses exec() extensively (when normally it can only be used once and then the program exits), it leads me to think you have something like this:
   Process.fork{ exec(cmd) }
which of course would spawn a lot of zombies. You need to detach the child process from the parent:
   pid = Process.fork{ exec(cmd) }
   Process.detach(pid)

Daniel

5

It sounds like your use cases are rather simple and really only need
to be focusing on `` and system().
In this case use `` when you need to capture the output and system()
otherwise.

Of course I'm assuming that your need is simple here.

The need is simple, I think. But the only difference between using
exec() and system() in this case is that the zombies now have a
different listing for CMD in ps.

On examining the code it looks as if these external processes require
something to be configured which is not configured. So it appears that
every one of these zombies is failing to even do anything.

···

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org

6

> I've got code somebody else wrote. The code uses exec() to invoke
> shell processes. It spawns zombies by the thousands.
>
> I need to redo this over the long term to stop using shell processes at
> all.

why? it's one of ruby's strengths that it can act as glue between
external components rather than having to code everything yourself.

Yeah, but given the userbase etc. it should be using BackgrounDRb
rather than external shell processes, if it needs external processes.
In fact, it didn't have such a need; the shell process was calling the
app itself via HTTP over curl, and the code was already within the
same app.

Yay rapid prototyping.

···

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org

7

Giles,

That is a bit odd, I suppose a snippet of the code that produces the zombies would be necessary to help further.

exec() would kill your ruby process and not return.
system() should execute the command then return to the ruby process with exit status in $?.

···

On Jul 05, 2007, at 16:10 , Giles Bowkett wrote:

The need is simple, I think. But the only difference between using
exec() and system() in this case is that the zombies now have a
different listing for CMD in ps.

On examining the code it looks as if these external processes require
something to be configured which is not configured. So it appears that
every one of these zombies is failing to even do anything.

--
Wayne E. Seguin
Sr. Systems Architect & Systems Admin
wayneseguin@gmail.com

8

In that case, it sounds like you could just comment out the code that's
trying to call the external processes, and you would immediately eliminate
the zombies without losing any functionality that you have at the
moment. :slight_smile: I'd probably start there.

Next figure out whether you really need those external processes to
run. If not then you're done; maybe the reason to run them has gone away
since this was first set up. If you do need them, then configure them so
they run properly in 'standalone' mode, then uncomment the original code and
chances are you won't have the same zombie problem anymore.

As for exec() vs. system(), the biggest difference is whether you want any
code after that call to get executed. Since exec replaces the current
process with the new one, any code after the exec() call will never get run.
If it's the last line of the program anyway, then this won't matter to you.

···

On 7/5/07, Giles Bowkett <gilesb@gmail.com> wrote:

> It sounds like your use cases are rather simple and really only need
> to be focusing on `` and system().
> In this case use `` when you need to capture the output and system()
> otherwise.
>
> Of course I'm assuming that your need is simple here.

The need is simple, I think. But the only difference between using
exec() and system() in this case is that the zombies now have a
different listing for CMD in ps.

On examining the code it looks as if these external processes require
something to be configured which is not configured. So it appears that
every one of these zombies is failing to even do anything.

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org

--
Wes Sheldahl
wes.sheldahl@gmail.com

9

> On examining the code it looks as if these external processes require
> something to be configured which is not configured. So it appears that
> every one of these zombies is failing to even do anything.
>
In that case, it sounds like you could just comment out the code that's
trying to call the external processes, and you would immediately eliminate
the zombies without losing any functionality that you have at the
moment. :slight_smile: I'd probably start there.

In fact, that's what I did. The code didn't do anything, so I deleted it.

Next figure out whether you really need those external processes to
run. If not then you're done; maybe the reason to run them has gone away
since this was first set up. If you do need them, then configure them so
they run properly in 'standalone' mode, then uncomment the original code and
chances are you won't have the same zombie problem anymore.

The external processes actually called the app itself, via curl.

The external processes are now considered evil in my book. Anything
which spawns zombies is evil; anything which calls itself recursively
via HTTP is also evil. Therefore, evil.

QED.

(Actually, one day we'll probably have apps that call themselves via
HTTP, and it'll be higher-order HTTP, and that'll be great, but for
now, evil.)

As for exec() vs. system(), the biggest difference is whether you want any
code after that call to get executed. Since exec replaces the current
process with the new one, any code after the exec() call will never get run.
If it's the last line of the program anyway, then this won't matter to you.

Well, it wasn't the last line of code, but it wasn't really being
called properly. Ordinarily I use system() rather than exec(), on the
assumption that in most cases I want the program to continue its end.

···

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org