Getting a stack trace of a running Ruby program

ruby-talk
1

Is there a way of displaying the stack trace of a running ruby program without killing it?

Java virtual machines(*) will show the stack trace(s) of a running program
if you send them a QUIT signal (or on windows <ctrl><break>) without stopping
the program. On Linux, the "pstack" program gives you the same capabilities
for C programs.

This is useful when a program runs for days and gets gradually slower. By
periodically sending QUIT signals you can essentally poll the program's
stack to see where it's getting slow.

Is there any way to do similar in Ruby? Even if I need to attach using
GDB, it would help me - but I'd need someone to point me to what data
structures I should look at with the debugger. Otherwise, I think it'd
be a really nice feature if, like java, a QUIT signal could show the
stack trace(s) of the existing nthread(s) without killing an application

    Ron

* http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/
* http://linuxcommand.org/man_pages/pstack1.html

2

I prefer to use real profiling tools instead of looking at a single stacktrace. In Java that would be -Xrunhprof, OptimizeIt et al. In Ruby you can use -r profile for profiling. Also, there's module Benchmark which is less intrusive than the profiler. Finally there's set_trace_func which can be used to generate call stacks for each thread and print them to console on a signal similarly to your Java approach.

HTH

    robert

···

Ron M <rm_rails@cheapcomplexdevices.com> wrote:

Is there a way of displaying the stack trace of a running ruby
program without killing it?
Java virtual machines(*) will show the stack trace(s) of a running
program if you send them a QUIT signal (or on windows <ctrl><break>) without
stopping the program. On Linux, the "pstack" program gives you the same
capabilities for C programs.

This is useful when a program runs for days and gets gradually
slower. By periodically sending QUIT signals you can essentally poll the
program's stack to see where it's getting slow.

Is there any way to do similar in Ruby? Even if I need to attach
using GDB, it would help me - but I'd need someone to point me to what data
structures I should look at with the debugger. Otherwise, I think
it'd be a really nice feature if, like java, a QUIT signal could show the
stack trace(s) of the existing nthread(s) without killing an
application

   Ron

*
Oracle Java Technologies | Oracle
* http://linuxcommand.org/man_pages/pstack1.html

3

Ruby Performance Validator and/or Ruby Bug Validator can do this.

        http://www.softwareverify.com

Stephen

···

In message <436BD294.7040005@cheapcomplexdevices.com>, Ron M <rm_rails@cheapcomplexdevices.com> writes

Is there a way of displaying the stack trace of a running ruby program
without killing it?

--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting

4

trap('QUIT') { puts caller.join("\n") }

Probably better to trap a signal other than QUIT, like USR1

···

On Nov 4, 2005, at 1:26 PM, Ron M wrote:

Is there a way of displaying the stack trace of a running ruby program without killing it?

Java virtual machines(*) will show the stack trace(s) of a running program
if you send them a QUIT signal (or on windows <ctrl><break>) without stopping
the program. On Linux, the "pstack" program gives you the same capabilities
for C programs.

--
Eric Hodel - drbrain@segment7.net - http://segment7.net
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

5

Ron M wrote:

Is there a way of displaying the stack trace of a running ruby program
without killing it?

Java virtual machines(*) will show the stack trace(s) of a running program
if you send them a QUIT signal (or on windows <ctrl><break>) without
stopping
the program. On Linux, the "pstack" program gives you the same
capabilities
for C programs.

This is useful when a program runs for days and gets gradually slower. By
periodically sending QUIT signals you can essentally poll the program's
stack to see where it's getting slow.

Is there any way to do similar in Ruby? Even if I need to attach using
GDB, it would help me - but I'd need someone to point me to what data
structures I should look at with the debugger. Otherwise, I think it'd
be a really nice feature if, like java, a QUIT signal could show the
stack trace(s) of the existing nthread(s) without killing an application

   Ron

* Oracle Java Technologies | Oracle
* http://linuxcommand.org/man_pages/pstack1.html

Yes, this is a very late comment to this thread, but as it happens I
needed something like this last week. Unfortunately none of the proposed
solutions in this thread would have worked for me, because I was
debugging a multithreaded ruby process. (I was debugging from inside the
process using irb while other threads ran.)

Robert Klemme's suggestion to fork and raise an exception is nice, but
only works an a single thread. A forked ruby process keeps only one
thread from the parent (unless something has changed since 1.8.2):

  threads = (0..3).map {Thread.new {sleep}}

  p Thread.list.size # ==> 5

  sleep 1

  fork do
    p Thread.list.size # ==> 1
  end

  sleep 1

In any case, #fork doesn't work on all platforms.

Eric Hodel's suggestion to trap some signal and use #caller is also
nice, but only gives you information about one thread (the main thread,
IIRC).

Robert also suggested set_trace_func. But that doesn't tell you where
your threads are sleeping, unless you've enabled the trace func before
the threads went to sleep. Keeping set_trace_func enabled can have
serious overhead costs.

It would be very nice to be able to ask a thread for its current backtrace.

···

--
      vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

6

Robert Klemme wrote:

Is there a way of displaying the stack trace of a running ruby
[like] Java QUIT signal
[or] the "pstack" program gives you the same capabilities for C programs.

I prefer to use real profiling tools instead of looking at a single stacktrace. In Java that would be -Xrunhprof, OptimizeIt et al. In Ruby you can use -r profile for profiling. Also, there's module Benchmark which is less intrusive than the profiler. Finally there's set_trace_func which can be used to generate call stacks for each thread and print them to console on a signal similarly to your Java approach.

HTH

These all work fine for relatively short-lived programs; but
do little to help in a progam that runs for days and only gets
slow after many hours of running.

On short runs (few hours) my program works fine (processes
dozens of records per second) but after half a day or so
it slows to taking a few seconds per record. "-r profile"
and set_trace_func are pretty intrusive and I fear my
program that runs for days would take weeks. Benchmark
has the disadvantage of needing to know what piece of code
to measure; rather than exposing the slow part.

But the biggest advantage in my mind of sampling instantaneous
snapshots of the stack is that it works on any program witout
modification (well, linux's ptrace is most useful if you didn't
strip signals); and it's almost totally non-intrusive, and does
not skew the timing results at all.

···

Ron M <rm_rails@cheapcomplexdevices.com> wrote:

7

Yes. From what I've seen of the internals the current implementation does not have this information, well not in a format that is retrievable anyway. I'd like to be proved wrong on this point!

Stephen

···

In message <4374FB36.3090403@path.berkeley.edu>, Joel VanderWerf <vjoel@path.berkeley.edu> writes

It would be very nice to be able to ask a thread for its current backtrace.

--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk/software.html
Computer Consultancy, Software Development
Windows C++, Java, Assembler, Performance Analysis, Troubleshooting

8

so send it a signal that makes it fork and dump core - then you can do

   gdb ruby core.1234

whamo.

-a

···

On Sat, 5 Nov 2005, Ron M wrote:

Robert Klemme wrote:

Ron M <rm_rails@cheapcomplexdevices.com> wrote:

Is there a way of displaying the stack trace of a running ruby
[like] Java QUIT signal
[or] the "pstack" program gives you the same capabilities for C programs.

I prefer to use real profiling tools instead of looking at a single stacktrace. In Java that would be -Xrunhprof, OptimizeIt et al. In Ruby you can use -r profile for profiling. Also, there's module Benchmark which is less intrusive than the profiler. Finally there's set_trace_func which can be used to generate call stacks for each thread and print them to console on a signal similarly to your Java approach.

HTH

These all work fine for relatively short-lived programs; but
do little to help in a progam that runs for days and only gets
slow after many hours of running.

On short runs (few hours) my program works fine (processes
dozens of records per second) but after half a day or so
it slows to taking a few seconds per record. "-r profile"
and set_trace_func are pretty intrusive and I fear my
program that runs for days would take weeks. Benchmark
has the disadvantage of needing to know what piece of code
to measure; rather than exposing the slow part.

But the biggest advantage in my mind of sampling instantaneous
snapshots of the stack is that it works on any program witout
modification (well, linux's ptrace is most useful if you didn't
strip signals); and it's almost totally non-intrusive, and does
not skew the timing results at all.

--

email :: ara [dot] t [dot] howard [at] noaa [dot] gov
phone :: 303.497.6469
anything that contradicts experience and logic should be abandoned.
-- h.h. the 14th dalai lama

===============================================================================

9

How about this?

Signal.trap "INT" do
  fork do
    ObjectSpace.each_object(Thread) do |th|
      th.raise Exception, "Stack Dump" unless Thread.current == th
    end
    raise Exception, "Stack Dump"
  end
end

Thread.new do
  10.times do
    puts "."
    sleep 5
  end
end

10.times do
  puts "."
  sleep 5
end

This is non intrusive.

    robert

···

Ron M <rm_rails@cheapcomplexdevices.com> wrote:

Robert Klemme wrote:

Ron M <rm_rails@cheapcomplexdevices.com> wrote:

Is there a way of displaying the stack trace of a running ruby
[like] Java QUIT signal
[or] the "pstack" program gives you the same capabilities for C
programs.

I prefer to use real profiling tools instead of looking at a single
stacktrace. In Java that would be -Xrunhprof, OptimizeIt et al. In
Ruby you can use -r profile for profiling. Also, there's module
Benchmark which is less intrusive than the profiler. Finally there's
set_trace_func which can be used to generate call stacks for each
thread and print them to console on a signal similarly to your Java
approach.

HTH

These all work fine for relatively short-lived programs; but
do little to help in a progam that runs for days and only gets
slow after many hours of running.

On short runs (few hours) my program works fine (processes
dozens of records per second) but after half a day or so
it slows to taking a few seconds per record. "-r profile"
and set_trace_func are pretty intrusive and I fear my
program that runs for days would take weeks. Benchmark
has the disadvantage of needing to know what piece of code
to measure; rather than exposing the slow part.

But the biggest advantage in my mind of sampling instantaneous
snapshots of the stack is that it works on any program witout
modification (well, linux's ptrace is most useful if you didn't
strip signals); and it's almost totally non-intrusive, and does
not skew the timing results at all.

10

Robert Klemme wrote:

How about this?
[cool code fragment omitted] This is non intrusive.

Thanks! As always I'm impressed with Ruby that
every time I find something I wanted from another
platform you guys (you and Eric Hodel both in this case)
give me a handful of lines of code that enable Ruby with
everything I had needed.