How to reclaim memory without GC.start

ruby-talk
1

Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
  stream_host_connection.write buffer
  stream_host_connection.flush
  # GC.start - too expensive
end

Thanks, Dave

2

Unfortunately, I don't know how to do it. If I did, I would include it in every single one of my programs. But you should have an extra '=' after buffer. It should read...
...
while buffer == file.read(512)
...

Note the double '='.

G'luck, and PLEASE pop an email to the list if you figure out how to accomplish it.
-------------------------------------------|
Nietzsche is my copilot

···

On Jul 16, 2007, at 5:30 PM, dtuttle1@gmail.com wrote:

Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
  stream_host_connection.write buffer
  stream_host_connection.flush
  # GC.start - too expensive
end

3

dtuttle1@gmail.com wrote:

Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
  stream_host_connection.write buffer
  stream_host_connection.flush
  # GC.start - too expensive
end

Someone please correct me if I'm wrong, but...

You don't need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration -- it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you're really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.

···

--
  Travis Warlick

  "Programming in Java is like dealing with your mom --
   it's kind, forgiving, and gently chastising.
   Programming in C++ is like dealing with a disgruntled
   girlfriend -- it's cold, unforgiving, and doesn't tell
   you what you've done wrong."

4

Note that IO#read supports a buffer argument:

buffer = ""
while file.read(512, buffer)
  ...
end

In that case it won't create a new string object for each call. It
will just modify buffer instead.

···

On Jul 16, 11:30 pm, "dtutt...@gmail.com" <dtutt...@gmail.com> wrote:

file = File.new file_path
while buffer = file.read(512)
  stream_host_connection.write buffer
  stream_host_connection.flush
  # GC.start - too expensive
end

5

No, Ari. I think you misread the program. There is an assignment to
buffer, not a comparison. The while loop will exit when
file.read(512) returns nil. I know why you thought it was a
comparison, though.

Todd

···

On 7/16/07, Ari Brown <ari@aribrown.com> wrote:

On Jul 16, 2007, at 5:30 PM, dtuttle1@gmail.com wrote:

> Hi - I have a memory and cpu-constrained embedded app. There's a long-
> running loop with a local object (buffer) which eats up memory. A
> GC.start on every iteration will reclaim it, but that eats the cpu.
>
> Is there a way to deallocate the memory used by the temporary objects?
>
> file = File.new file_path
> while buffer = file.read(512)
> stream_host_connection.write buffer
> stream_host_connection.flush
> # GC.start - too expensive
> end

Unfortunately, I don't know how to do it. If I did, I would include
it in every single one of my programs. But you should have an extra
'=' after buffer. It should read...
...
while buffer == file.read(512)

6

Travis D Warlick Jr wrote the following on 17.07.2007 01:31 :

  

Hi - I have a memory and cpu-constrained embedded app. There's a long-
running loop with a local object (buffer) which eats up memory. A
GC.start on every iteration will reclaim it, but that eats the cpu.

Is there a way to deallocate the memory used by the temporary objects?

file = File.new file_path
while buffer = file.read(512)
  stream_host_connection.write buffer
  stream_host_connection.flush
  # GC.start - too expensive
end
    
Someone please correct me if I'm wrong, but...

You don't need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration -- it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you're really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.
  
Actually, as variable affectation is done by reference and the *content* of buffer is originally duplicated by File#read memory usage is not likely to change linearly with the value passed to File#read...

You can still play with different values passed to it and expect some change of behaviour (ie: depending on the actual amount of memory allocated for each read, you may or may not hit a sweat spot in the GC heuristics).

My advise would be to do larger reads and GC.start less often:

read_amount = 65536
read_count = 0
loops_between_gcs = 100
while buffer = file.read(read_amount)
  stream_host_connection.write buffer
  stream_host_connection.flush
  GC.start if ((read_count += 1) % loops_between_gcs) == 0
end

But note that if stream_host_connection.flush doesn't actually wait until the buffer isn't referenced anymore anywhere in your code (or in some other process sitting on the same system as a matter of fact), calling GC.start probably won't have any effect.

Lionel

···

dtuttle1@gmail.com wrote:

7

The object is being overwritten, but the old copies aren't being
garbage collected. I know this because the memory does stay constant
if I call GC.start on every iteration.
I think I'll go with a c extension for this small part of the code.

Thanks, Dave

···

On Jul 16, 4:31 pm, Travis D Warlick Jr <warli...@operissystems.com> wrote:

dtutt...@gmail.com wrote:
> Hi - I have a memory and cpu-constrained embedded app. There's a long-
> running loop with a local object (buffer) which eats up memory. A
> GC.start on every iteration will reclaim it, but that eats the cpu.

> Is there a way to deallocate the memory used by the temporary objects?

> file = File.new file_path
> while buffer = file.read(512)
> stream_host_connection.write buffer
> stream_host_connection.flush
> # GC.start - too expensive
> end

Someone please correct me if I'm wrong, but...

You don't need to deallocate buffer every time because it is not getting
duplicated at the beginning of every iteration -- it is being
overwritten, so the memory usage for buffer will always stay at 512
bytes (plus the static overhead for the object).

Deallocating it every time would actually increase CPU usage because you
would be deleting and recreating the buffer object at the beginning of
every iteration.

If you're really constrained on memory, you might want to consider
decreasing the amount that you read at every iteration. However,
remember that you increase your CPU usage by the same factor that you
decrease your memory usage in this case.

--
  Travis Warlick

  "Programming in Java is like dealing with your mom --
   it's kind, forgiving, and gently chastising.
   Programming in C++ is like dealing with a disgruntled
   girlfriend -- it's cold, unforgiving, and doesn't tell
   you what you've done wrong."

8

buffer = ""
while file.read(512, buffer)
  ...
end

You could save one more line:

while file.read(512, buffer||="")
...
end

gegroet,
Erik V. - http://www.erikveen.dds.nl/

9

Lionel Bouton wrote:

Actually, as variable affectation is done by reference

I knew that... woops... **beats head on keyboard**

However, A chance to redeem myself: I remembered that IO#read returns a
string, so change the while to:

    while buffer[0...512] = file.read(512)

That should overwrite the contents of the buffer

···

--
  Travis Warlick

  "Programming in Java is like dealing with your mom --
   it's kind, forgiving, and gently chastising.
   Programming in C++ is like dealing with a disgruntled
   girlfriend -- it's cold, unforgiving, and doesn't tell
   you what you've done wrong."

10

Yups. That is correct. My mistake, and thanks for pointing it out! At least now I know I can do assignment in my 'while' statements. Right?

~ Ari
English is like a pseudo-random number generator - there are a bajillion rules to it, but nobody cares.

···

On Jul 16, 2007, at 7:22 PM, Todd Benson wrote:

No, Ari. I think you misread the program. There is an assignment to
buffer, not a comparison. The while loop will exit when
file.read(512) returns nil. I know why you thought it was a
comparison, though.

11

Travis D Warlick Jr wrote:

Lionel Bouton wrote:

Actually, as variable affectation is done by reference

I knew that... woops... **beats head on keyboard**

However, A chance to redeem myself: I remembered that IO#read returns a
string, so change the while to:

    while buffer[0...512] = file.read(512)

That should overwrite the contents of the buffer

Do you mean file.read(512, buffer) ?

---------------------------------------------------------------- IO#read
      ios.read([length [, buffer]]) => string, buffer, or nil

···

------------------------------------------------------------------------
      Reads at most length bytes from the I/O stream, or to the end of
      file if length is omitted or is nil. length must be a non-negative
      integer or nil. If the optional buffer argument is present, it
      must reference a String, which will receive the data.

      At end of file, it returns nil or "" depend on length. ios.read()
      and ios.read(nil) returns "". ios.read(positive-integer) returns
      nil.

         f = File.new("testfile")
         f.read(16) #=> "This is line one"

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

12

Joel VanderWerf wrote:

Travis D Warlick Jr wrote:

Lionel Bouton wrote:

    while buffer[0...512] = file.read(512)

Do you mean file.read(512, buffer) ?

Ahh, yes. That would be an much prettier way to do it.

    while file.read(512, buffer)

···

--
  Travis Warlick

  "Programming in Java is like dealing with your mom --
   it's kind, forgiving, and gently chastising.
   Programming in C++ is like dealing with a disgruntled
   girlfriend -- it's cold, unforgiving, and doesn't tell
   you what you've done wrong."

13

Travis D Warlick Jr wrote:

Joel VanderWerf wrote:

Travis D Warlick Jr wrote:

Lionel Bouton wrote:

    while buffer[0...512] = file.read(512)

Do you mean file.read(512, buffer) ?

Ahh, yes. That would be an much prettier way to do it.

    while file.read(512, buffer)

It's not just prettier. With an assignment like

buffer[0...512] = file.read(512)

there is an intermediate string that instantly becomes garbage.

···

--
       vjoel : Joel VanderWerf : path berkeley edu : 510 665 3407

14

There is a better way to do it:

buffer = "." * 1024
while STDIN.read(1024, buffer)
  print buffer
end

So the original code should be rewritten as

File.open file_path do |file|
  buffer = "." * 512
  while file.read(512, buffer)
    stream_host_connection.write buffer
    stream_host_connection.flush
  end
end

The main point being here to reuse the buffer. No C extension needed.
See class IO - RDoc Documentation

Kind regards

robert

···

2007/7/17, Joel VanderWerf <vjoel@path.berkeley.edu>:

Travis D Warlick Jr wrote:
> Joel VanderWerf wrote:
>> Travis D Warlick Jr wrote:
>>> Lionel Bouton wrote:
>>>
>>> while buffer[0...512] = file.read(512)
>>>
>> Do you mean file.read(512, buffer) ?
>
> Ahh, yes. That would be an much prettier way to do it.
>
> while file.read(512, buffer)

It's not just prettier. With an assignment like

buffer[0...512] = file.read(512)

there is an intermediate string that instantly becomes garbage.

15

It works perfectly! Thanks everyone!

···

On Jul 17, 2:51 am, "Robert Klemme" <shortcut...@googlemail.com> wrote:

2007/7/17, Joel VanderWerf <vj...@path.berkeley.edu>:

> Travis D Warlick Jr wrote:
> > Joel VanderWerf wrote:
> >> Travis D Warlick Jr wrote:
> >>> Lionel Bouton wrote:

> >>> while buffer[0...512] = file.read(512)

> >> Do you mean file.read(512, buffer) ?

> > Ahh, yes. That would be an much prettier way to do it.

> > while file.read(512, buffer)

> It's not just prettier. With an assignment like

> buffer[0...512] = file.read(512)

> there is an intermediate string that instantly becomes garbage.

There is a better way to do it:

buffer = "." * 1024
while STDIN.read(1024, buffer)
  print buffer
end

So the original code should be rewritten as

File.open file_path do |file|
  buffer = "." * 512
  while file.read(512, buffer)
    stream_host_connection.write buffer
    stream_host_connection.flush
  end
end

The main point being here to reuse the buffer. No C extension needed.
Seehttp://www.ruby-doc.org/core/classes/IO.html#M002295

Kind regards

robert

16

buffer = "." * 1024
while STDIN.read(1024, buffer)
  print buffer
end

What's the point of initializing the buffer with 1024 characters,
instead of just initializing it with an empty string? I must be
missing something...

Thanks.

gegroet,
Erik V.

17

> buffer = "." * 1024
> while STDIN.read(1024, buffer)
> print buffer
> end

What's the point of initializing the buffer with 1024 characters,
instead of just initializing it with an empty string? I must be
missing something...

Thanks.

I think the idea was to do the allocation upfront instead of when witting to
the string. But since that will only happen once, (on the first read) it
seems unnecessary. But I could be wrong.

gegroet,

···

On 8/4/07, Erik Veenstra <erikveen@gmail.com> wrote:

Erik V.

18

Finally! I've been waiting for someone to resurrect this thread.

Ok. I've decided that if a program consumes ALL of a computer's available memory (free, inactive), then GC.start is run, it will be able to free all that RAM.

I have done several scientific tests with this on World of Warcraft (heh. work on WoW) and image rendering.

So what's the best way to consume all of a computer's RAM? I've been toying with the idea of massive variable generation, but i think that might be a bit difficult.

So what do you all think?
---------------------------------------------------------------|
~Ari
"I don't suffer from insanity. I enjoy every minute of it" --1337est man alive

19

I believe you are right. I probably rather did it for documentary
reasons but in fact it's not necessary. There would only be a small
but likely unnoticeable advantage if for some reason #read would not
read up to limit chars right away. Then there might be multiple
reallocations (i.e. until the buffer has reached the limit). But I
guess this is not very likely, at least not with file IO. Situation
might be different for sockets.

Kind regards

robert

···

2007/8/6, Logan Capaldo <logancapaldo@gmail.com>:

On 8/4/07, Erik Veenstra <erikveen@gmail.com> wrote:
>
> > buffer = "." * 1024
> > while STDIN.read(1024, buffer)
> > print buffer
> > end
>
> What's the point of initializing the buffer with 1024 characters,
> instead of just initializing it with an empty string? I must be
> missing something...
>
> Thanks.

I think the idea was to do the allocation upfront instead of when witting to
the string. But since that will only happen once, (on the first read) it
seems unnecessary. But I could be wrong.

20

I'm not sure what you're after. Consuming memory is easy. For example:

a=;a << " "*1024 while true

Or, a bit more involved

Chain = Struct.new :data, :next
c = nil
c = Chain.new("."*1024, c) while true

Kind regards

robert

···

2007/8/6, Ari Brown <ari@aribrown.com>:

Finally! I've been waiting for someone to resurrect this thread.

Ok. I've decided that if a program consumes ALL of a computer's
available memory (free, inactive), then GC.start is run, it will be
able to free all that RAM.

I have done several scientific tests with this on World of Warcraft
(heh. work on WoW) and image rendering.

So what's the best way to consume all of a computer's RAM? I've been
toying with the idea of massive variable generation, but i think that
might be a bit difficult.

So what do you all think?