I want to profile the code. Is your test suite the one bundled with
testunit?
yes. I've seen this big slowdown only with bz2 : libbzip2 internally use
big buffers to decompress the data, with bz2 these buffers are managed by
ruby (ruby_xmalloc). This means that each time I call ::new, the GC is
called.
This is only how I can explain this slowdown with bz2
yes. I’ve seen this big slowdown only with bz2 : libbzip2 internally use
big buffers to decompress the data, with bz2 these buffers are managed by
ruby (ruby_xmalloc). This means that each time I call ::new, the GC is
called.
I guess this issue and [ruby-talk:59662] concern with following
change. GC only knows “big buffers” were allocated, but
doesn’t know whether they’re refered by live objects, so the
live memories amount would be estimated too small than the
allocated.
* gc.c (ruby_xmalloc, ruby_xrealloc): restrict total allocation
size according to memories consumed by live objects.
[ruby-dev:18482]
* gc.c (gc_sweep): estimate how live objects consume memories.
I want to profile the code. Is your test suite the one bundled with
testunit?
yes. I’ve seen this big slowdown only with bz2 : libbzip2 internally use
big buffers to decompress the data, with bz2 these buffers are managed by
ruby (ruby_xmalloc). This means that each time I call ::new, the GC is
called.
This is only how I can explain this slowdown with bz2
Remember the “speed differences” I posted a while ago?
Same issue.
gc_sweep() got called for each allocation of 100kB, meaning it was ran
about 100 times more often for 1.7.3 than for 1.6.8. If I change the
allocation size to 100 bytes (instead of 100 kB) 1.7.3 speeds up to match
1.6.8
Suggestion to fix: the GC should not run, unless a threshold on the amount
of allocations is crossed. (But my theory on gc is weak, so I am probably
missing some things here.)
I guess this issue and [ruby-talk:59662] concern with following
change. GC only knows "big buffers" were allocated, but
doesn't know whether they're refered by live objects, so the
live memories amount would be estimated too small than the
allocated.
Well, I have also changed Data_Make_Struct to be sure that it call also
malloc() rather than ruby_xmalloc() and it seems to have a problem with
the GC. It is called more often
* gc.c (ruby_xmalloc, ruby_xrealloc): restrict total allocation
size according to memories consumed by live objects.
[ruby-dev:18482]
* gc.c (gc_sweep): estimate how live objects consume memories.
Yes, this is this that it don't like.
With one my test (reader.rb) it never call the GC with 1.6.8, but call it
many times with 1.8 (this is a version of bz2 with malloc() for
Data_Make_Struct() and the libbzip2 buffers).
gc_sweep() got called for each allocation of 100kB, meaning it was ran
about 100 times more often for 1.7.3 than for 1.6.8. If I change the
allocation size to 100 bytes (instead of 100 kB) 1.7.3 speeds up to match
1.6.8
Well, the problem is easy to understand if you look the source.
The patch is described
* gc.c (ruby_xmalloc, ruby_xrealloc): restrict total allocation
size according to memories consumed by live objects.
[ruby-dev:18482]
* gc.c (gc_sweep): estimate how live objects consume memories.
Unfortunately this is wrong : ruby don't estimate the size of *all*
objects but only the size of *some* objects (like T_OBJECT, T_ARRAY, ...)
it don't even try to correct this value to have a real estimation of the
memory, and even if it will try do it it will have a problem.
The problem is that ruby has selected a subpopulation and this
subpopulation is not representative of all objects.
