Segmentation fault, proc, eval, long string

ruby-talk
21

Haha. Oops. You're right:

wilson@metaclass:~$ ruby boom.rb
the_string length: 186747
[[:module,
  :SomeModule,
  [:scope,
   [:cvdecl,
    :@@pr,
    [:iter,
     [:call, [:const, :Proc], :new],
     nil,
     [:block,
      [:dasgn_curr,
       :thing,
       [:dasgn_curr,
        :v0,
        [:dasgn_curr,
         :v1,
         [:dasgn_curr,
          :v2,
          [:dasgn_curr,
           :v3,
           [:dasgn_curr,
            :v4,
            [:dasgn_curr,
             :v5,
/usr/lib/ruby/1.8/prettyprint.rb:344:in `deq': stack level too deep
(SystemStackError)
        from /usr/lib/ruby/1.8/prettyprint.rb:343:in `deq'
        from /usr/lib/ruby/1.8/prettyprint.rb:171:in `break_outmost_groups'
        from /usr/lib/ruby/1.8/prettyprint.rb:197:in `text'
        from /usr/lib/ruby/1.8/pp.rb:245:in `pretty_print'
        from /usr/lib/ruby/1.8/pp.rb:126:in `pp'
        from /usr/lib/ruby/1.8/prettyprint.rb:224:in `group'
        from /usr/lib/ruby/1.8/prettyprint.rb:247:in `nest'
        from /usr/lib/ruby/1.8/prettyprint.rb:223:in `group'
         ... 294 levels...
        from /usr/lib/ruby/1.8/pp.rb:69:in `pp'
        from /usr/lib/ruby/1.8/pp.rb:52:in `pp'
        from /usr/lib/ruby/1.8/pp.rb:51:in `pp'
        from boom.rb:45

···

On 12/1/06, Pit Capitain <pit@capitain.de> wrote:

Wilson Bilkovich schrieb:
> On 12/1/06, Pit Capitain <pit@capitain.de> wrote:
>> Bob, you can use parsetree to dump the AST of the generated proc. I'm
>> sure you'll see the deep nesting of the nodes.
>
> I just tried this, and here's what it gave me (for a smaller N, so the
> whole process doesn't crash. Should show the same structure no matter
> what N is, though)
>
> (...)

Wilson, this is not the dump of the generated proc. You have to pass the
contents of @@proc to ParseTree.

22

I get a segfault too on Ubuntu Edgy x86_64.

efine@ender:/tmp$ uname -a
Linux ender 2.6.17-10-generic #2 SMP Fri Oct 13 15:34:39 UTC 2006 x86_64
GNU/Linux

efine@ender:/tmp$ ruby segfault.rb
the_string length: 186749
Segmentation fault (core dumped)

I don't want to insult anyone's intelligence by stating what could be to
many the obvious, but I'd like to make an observation.

An iterative version of the algorithm might or might not solve the
problem; it all depends how much data has to be stacked on each
iteration (e.g. closures), and where/how it is stacked. If malloc() is
used to create a stack for use with the iterative code, the system will
practically have to run out of VM before the process dies, but using
malloc instead of hardware stack allocation may create performance
problems unless special care is taken, which could be a lot of work.

Is it worth the effort to cater for this special case, given the ulimit
-s workaround? Incidentally, on my system, the default stack size is
only 8MB. That's not a lot these days (given memory capacities these
days).

Just a thought.

···

--
Posted via http://www.ruby-forum.com/.

23

Hi,

>> Thank you for the report. Your script helped. Could you check if the
>> attached patch work for you?
>
>Thanks for the patch. I applied it and tried a few things with it.
>The test program I provided does work now, so I think this is solving
>the problem. On the other hand, when I try to run the application
>that needs it, debian is killing it and I can't see why. The
>installation I'm using, as it happens, doesn't have any VM configured
>and this has been causing some difficulty recently. Could this patch
>cause a lot of memory to be allocated rapidly? If not then there's
>something else going on that I'll have to look into (and it is likely
>a completely different problem).

The patch does rather decrease the amount of memory that Ruby use.
But the original program seems to use tens of thousands of in-block
variables, which themselves consumes more memory than plain local
variables or arrays. So by avoiding segmentation fault, it turns out
to kick the out-of-memory killer of linux kernel.

If it's possible, I'd recommend you to reduce these local variables.

This is consistent with what I'm seeing. Just before you posted your patch, I made a couple of changes to the application and also replaced the Proc with a method. The changes to the application would reduce the maximum number of local variables by typically one third. The use of a method rather than a Proc avoided the nested parsing and subsequent stack overflow (and is many times faster for large cases). Even with that the application barely fits into the available memory. So if the Proc's local variables cause even a relatively small proportional increase in memory the application is going to get itself killed.

Anyway, with the combination of all of this, and a better linux configuration I think I'll be past this.

Thanks again to everyone for your help.

Cheers,
Bob

···

On 3-Dec-06, at 8:59 PM, Yukihiro Matsumoto wrote:

In message "Re: Segmentation fault, proc, eval, long string > [Reproduced]" > on Sun, 3 Dec 2006 22:51:38 +0900, Bob Hutchison > <hutch@recursive.ca> writes:

              matz.

----
Bob Hutchison -- blogs at <http://www.recursive.ca/hutch/&gt;
Recursive Design Inc. -- <http://www.recursive.ca/&gt;
Raconteur -- <http://www.raconteur.info/&gt;
xampl for Ruby -- <http://rubyforge.org/projects/xampl/&gt;

24

OK. Wow.
I moved the code outside of a proc, and into a bare module, and re-ran
parse_tree on it.
[[:module,
  :SomeModule,
  [:scope,
   [:block,
    [:lasgn, :thing, [:zarray]],
    [:lasgn, :v0, [:array, [:lit, 0]]],
    [:lasgn, :v1, [:array, [:lit, 1]]],
    [:lasgn, :v2, [:array, [:lit, 2]]],
    [:lasgn, :v3, [:array, [:lit, 3]]],
    [:lasgn, :v4, [:array, [:lit, 4]]],
    [:lasgn, :v5, [:array, [:lit, 5]]],
    [:lasgn, :v6, [:array, [:lit, 6]]],
    [:lasgn, :v7, [:array, [:lit, 7]]],
    [:lasgn, :v8, [:array, [:lit, 8]]],
    [:lasgn, :v9, [:array, [:lit, 9]]],
    [:lasgn, :v10, [:array, [:lit, 10]]],
    [:lasgn, :v11, [:array, [:lit, 11]]],
    [:lasgn, :v12, [:array, [:lit, 12]]],
    [:lasgn, :v13, [:array, [:lit, 13]]],
    [:lasgn, :v14, [:array, [:lit, 14]]],
    [:lasgn, :v15, [:array, [:lit, 15]]],
    [:lasgn, :v16, [:array, [:lit, 16]]],
    [:lasgn, :v17, [:array, [:lit, 17]]],
    [:lasgn, :v18, [:array, [:lit, 18]]],
    [:lasgn, :v19, [:array, [:lit, 19]]],
    [:lasgn, :v20, [:array, [:lit, 20]]],
    [:lasgn, :v21, [:array, [:lit, 21]]],
    [:lasgn, :v22, [:array, [:lit, 22]]],
    [:lasgn, :v23, [:array, [:lit, 23]]],
    [:lasgn, :v24, [:array, [:lit, 24]]],
    [:lasgn, :v25, [:array, [:lit, 25]]],
    [:lasgn, :v26, [:array, [:lit, 26]]],
    [:lasgn, :v27, [:array, [:lit, 27]]],
    [:lasgn, :v28, [:array, [:lit, 28]]],
    [:lasgn, :v29, [:array, [:lit, 29]]],
    [:lasgn, :v30, [:array, [:lit, 30]]],
    [:lasgn, :v31, [:array, [:lit, 31]]],
    [:lasgn, :v32, [:array, [:lit, 32]]],
    [:lasgn, :v33, [:array, [:lit, 33]]],
    [:lasgn, :v34, [:array, [:lit, 34]]],
    [:lasgn, :v35, [:array, [:lit, 35]]],
    [:lasgn, :v36, [:array, [:lit, 36]]],
    [:lasgn, :v37, [:array, [:lit, 37]]],
    [:lasgn, :v38, [:array, [:lit, 38]]],
    [:lasgn, :v39, [:array, [:lit, 39]]],
    [:lasgn, :v40, [:array, [:lit, 40]]],
    [:lasgn, :v41, [:array, [:lit, 41]]],
    [:lasgn, :v42, [:array, [:lit, 42]]],
    [:lasgn, :v43, [:array, [:lit, 43]]],
    [:lasgn, :v44, [:array, [:lit, 44]]],
    [:lasgn, :v45, [:array, [:lit, 45]]],
    [:lasgn, :v46, [:array, [:lit, 46]]],
    [:lasgn, :v47, [:array, [:lit, 47]]],
    [:lasgn, :v48, [:array, [:lit, 48]]],
    [:lasgn, :v49, [:array, [:lit, 49]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v0]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v1]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v2]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v3]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v4]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v5]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v6]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v7]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v8]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v9]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v10]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v11]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v12]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v13]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v14]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v15]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v16]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v17]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v18]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v19]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v20]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v21]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v22]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v23]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v24]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v25]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v26]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v27]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v28]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v29]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v30]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v31]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v32]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v33]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v34]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v35]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v36]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v37]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v38]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v39]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v40]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v41]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v42]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v43]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v44]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v45]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v46]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v47]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v48]]],
    [:call, [:lvar, :thing], :<<, [:array, [:lvar, :v49]]],
    [:lvar, :thing]]]]]

It looks like the parser treats things very differently inside a Proc
definition.

···

On 12/1/06, Wilson Bilkovich <wilsonb@gmail.com> wrote:

On 12/1/06, Pit Capitain <pit@capitain.de> wrote:
> Wilson Bilkovich schrieb:
> > On 12/1/06, Pit Capitain <pit@capitain.de> wrote:
> >> Bob, you can use parsetree to dump the AST of the generated proc. I'm
> >> sure you'll see the deep nesting of the nodes.
> >
> > I just tried this, and here's what it gave me (for a smaller N, so the
> > whole process doesn't crash. Should show the same structure no matter
> > what N is, though)
> >
> > (...)
>
> Wilson, this is not the dump of the generated proc. You have to pass the
> contents of @@proc to ParseTree.
>

Haha. Oops. You're right:

wilson@metaclass:~$ ruby boom.rb
the_string length: 186747
[[:module,
  :SomeModule,
  [:scope,
   [:cvdecl,
    :@@pr,
    [:iter,
     [:call, [:const, :Proc], :new],
     nil,
     [:block,
      [:dasgn_curr,
       :thing,
       [:dasgn_curr,
        :v0,
        [:dasgn_curr,
         :v1,
         [:dasgn_curr,
          :v2,
          [:dasgn_curr,
           :v3,
           [:dasgn_curr,
            :v4,
            [:dasgn_curr,
             :v5,
/usr/lib/ruby/1.8/prettyprint.rb:344:in `deq': stack level too deep
(SystemStackError)
        from /usr/lib/ruby/1.8/prettyprint.rb:343:in `deq'
        from /usr/lib/ruby/1.8/prettyprint.rb:171:in `break_outmost_groups'
        from /usr/lib/ruby/1.8/prettyprint.rb:197:in `text'
        from /usr/lib/ruby/1.8/pp.rb:245:in `pretty_print'
        from /usr/lib/ruby/1.8/pp.rb:126:in `pp'
        from /usr/lib/ruby/1.8/prettyprint.rb:224:in `group'
        from /usr/lib/ruby/1.8/prettyprint.rb:247:in `nest'
        from /usr/lib/ruby/1.8/prettyprint.rb:223:in `group'
         ... 294 levels...
        from /usr/lib/ruby/1.8/pp.rb:69:in `pp'
        from /usr/lib/ruby/1.8/pp.rb:52:in `pp'
        from /usr/lib/ruby/1.8/pp.rb:51:in `pp'
        from boom.rb:45

25

Hi,

···

In message "Re: Segmentation fault, proc, eval, long string [Reproduced]" on Mon, 4 Dec 2006 21:50:22 +0900, Bob Hutchison <hutch@recursive.ca> writes:

This is consistent with what I'm seeing. Just before you posted your
patch, I made a couple of changes to the application and also
replaced the Proc with a method. The changes to the application would
reduce the maximum number of local variables by typically one third.
The use of a method rather than a Proc avoided the nested parsing and
subsequent stack overflow (and is many times faster for large cases).
Even with that the application barely fits into the available memory.
So if the Proc's local variables cause even a relatively small
proportional increase in memory the application is going to get
itself killed.

Anyway, with the combination of all of this, and a better linux
configuration I think I'll be past this.

Good to know that. For your information, YARV does not have this
problem by better in-block variable handling. Another reason to wait
for YARV.

              matz.

26

*That's* what I want! What's so special about a proc?

Cheers,
Bob

···

On 1-Dec-06, at 11:21 AM, Wilson Bilkovich wrote:

OK. Wow.
I moved the code outside of a proc, and into a bare module, and re-ran
parse_tree on it.
[[:module,
:SomeModule,
[:scope,
  [:block,
   [:lasgn, :thing, [:zarray]],
   [:lasgn, :v0, [:array, [:lit, 0]]],
   [:lasgn, :v1, [:array, [:lit, 1]]],
   [:lasgn, :v2, [:array, [:lit, 2]]],
   [:lasgn, :v3, [:array, [:lit, 3]]],
   [:lasgn, :v4, [:array, [:lit, 4]]],
   [:lasgn, :v5, [:array, [:lit, 5]]],

----
Bob Hutchison -- blogs at <http://www.recursive.ca/hutch/&gt;
Recursive Design Inc. -- <http://www.recursive.ca/&gt;
Raconteur -- <http://www.raconteur.info/&gt;
xampl for Ruby -- <http://rubyforge.org/projects/xampl/&gt;

27

Apparently, they are specially buggy. :slight_smile:

···

On 12/1/06, Bob Hutchison <hutch@recursive.ca> wrote:

On 1-Dec-06, at 11:21 AM, Wilson Bilkovich wrote:

> OK. Wow.
> I moved the code outside of a proc, and into a bare module, and re-ran
> parse_tree on it.
> [[:module,
> :SomeModule,
> [:scope,
> [:block,
> [:lasgn, :thing, [:zarray]],
> [:lasgn, :v0, [:array, [:lit, 0]]],
> [:lasgn, :v1, [:array, [:lit, 1]]],
> [:lasgn, :v2, [:array, [:lit, 2]]],
> [:lasgn, :v3, [:array, [:lit, 3]]],
> [:lasgn, :v4, [:array, [:lit, 4]]],
> [:lasgn, :v5, [:array, [:lit, 5]]],

*That's* what I want! What's so special about a proc?

Cheers,
Bob