Memoize and yaml

ruby-talk
1

Hello,

I am using the "memoize" module to eliminate having to redo complex,
time-consuming calculations. It works great. However, when I yamlize
my "memoized" object off to the database and then retrieve it back,
all the memoizing is lost.

My Ruby skills are still in development. In addition to normal
yamlizing, how could one also yamlize the memoize information, so that
yamlizing and back retains the methods that are to be memoized and
also retains any prior saved (memoized) calculations?

The test below on Foo illustrates what I am trying to do. Also below
is the whole Memoize module.

Any tips or pointers appreciated. Thanks!

--Brian Buckley

require 'test/unit'
require 'yaml'
require 'memoize'

class TestMemoizeAndYaml < Test::Unit::TestCase
  class Foo
    include Memoize
    attr_reader :x, :y
    def initialize(x, y)
      @x, @y = x, y
    end
    def calc1(a, b)
      sleep 1 # fakes big, complex calc
      ans = calc2 + a + b
    end
    def calc2(c = 0)
      sleep 1 # another big, complex calc
      ans = x + y
    end
  end

  def exercise_foo(foo)
    3.times{ foo.calc1 10,20 }
  end

  def test_memoize_and_yaml

    foo = Foo.new(1, 2) # first a foo without memoizing
    exercise_foo(foo) # without memoizing exercise_foo takes 6 seconds
- good

    # now memoize it
    foo.memoize :calc1 # memoize calc1
    foo.memoize :calc2 # and a 2nd method
    exercise_foo(foo) # good - now it takes only 2 secs because of the memoizing
    exercise_foo(foo) # and now no seconds because of the memoizing

    foo = YAML::load(foo.to_yaml) # to the database and back
    exercise_foo(foo) # but now exercise_foo is taking 6 secs again
    # how can I retain memoized state?
    # need to retain both the methods and the hashes of calculation histories
  end
end

#FYI... here is the Memoize module
module Memoize
   MEMOIZE_VERSION = "1.1.0"
   def memoize(name)
      meth = method(name)
      cache = {}
      (class << self; self; end).class_eval do
         define_method(name) do |*args|
            cache.has_key?(args) ? cache[args] : cache[args] ||=
meth.call(*args)
         end
      end
      cache
   end
end

2

Brian Buckley wrote:

I am using the "memoize" module to eliminate having to redo complex,
time-consuming calculations. It works great. However, when I yamlize
my "memoized" object off to the database and then retrieve it back,
all the memoizing is lost. [...]

#FYI... here is the Memoize module
module Memoize
   MEMOIZE_VERSION = "1.1.0"
   def memoize(name)
      meth = method(name)
      cache = {}
      (class << self; self; end).class_eval do
         define_method(name) do |*args|
            cache.has_key?(args) ? cache[args] : cache[args] ||=
meth.call(*args)
         end
      end
      cache
   end
end

This memoize module stores the cache in a local variable that can be referenced from within the memoize method wrapper.

It also memoizes the methods at the instance level meaning that after storing and loading an instance the methods won't be memoized anymore.

Here's a version that stores the cache in an instance variable and that replaces the method definitions at class definition time:

module Memoize
   def memoize(name)
     name = name.to_sym
     old_method = instance_method(name)
     remove_method(name)

     define_method(name) do |*args|
       @cache ||= {}
       signature = [name] + args

       if @cache.include?(signature) then
         @cache[signature]
       else
         @cache[signature] = old_method.bind(self).call(*args)
       end
     end
   end
end

Please note that you can use neither of these memoize() methods for memoizing methods that can take a block argument as it will be dropped.

Oh, and if you call a memoized method with an argument that can't be serialized by YAML this definition will probably cause trouble.

3

     old_method = instance_method(name)

I see the intent. I have not worked at it yet but I dropped your code
in as you wrote it and I am getting a NoMethodError: undefined method
`instance_method' on the above line.

--Brian

4

module Memoize
   def memoize(name)
     name = name.to_sym
     old_method = instance_method(name)
     remove_method(name)

     define_method(name) do |*args|
       @cache ||= {}
       signature = [name] + args

       if @cache.include?(signature) then
         @cache[signature]
       else
         @cache[signature] = old_method.bind(self).call(*args)
       end
     end
   end
end

Two questions of this solution. (It works and is what I need now so
I'm just looking to round out Ruby skills.)

1 Why is the 'remove_method(name)' line necessary? Does not the next
define_method call get rid of the method anyway by overriding?

2 The module requires use of 'extend' instead of 'include'. Is there
a comparable 'include' solution, and (if there is) would it be
preferred because, for example, one could not use this solution on a
class that already extends something else?

Thanks!

Brian Buckley

5

Brian Buckley wrote:

    old_method = instance_method(name)

I see the intent. I have not worked at it yet but I dropped your code
in as you wrote it and I am getting a NoMethodError: undefined method
`instance_method' on the above line.

Make sure to use the code on a module or class:

class Foo
   extend Memoize

   def foo(x)
     puts "Calculating #{x} ** 32"
     x ** 32
   end
   memoize :foo
end

6

Brian Buckley wrote:

module Memoize
  def memoize(name)
    name = name.to_sym
    old_method = instance_method(name)
    remove_method(name)

    define_method(name) do |*args|
      @cache ||= {}
      signature = [name] + args

      if @cache.include?(signature) then
        @cache[signature]
      else
        @cache[signature] = old_method.bind(self).call(*args)
      end
    end
  end
end

Two questions of this solution. (It works and is what I need now so
I'm just looking to round out Ruby skills.)

1 Why is the 'remove_method(name)' line necessary? Does not the next
define_method call get rid of the method anyway by overriding?

Overriding methods usually produces warnings. It's best to remove the methods first.

2 The module requires use of 'extend' instead of 'include'. Is there
a comparable 'include' solution, and (if there is) would it be
preferred because, for example, one could not use this solution on a
class that already extends something else?

Actually, I think this is the cleanest solution. In theory you could also offer a custom append_features() which defines the methods on the class or module including Memoize itself. That's more complex and not as easy to understand, though, in my opinion.

7

class Foo
   extend Memoize

   def foo(x)
     puts "Calculating #{x} ** 32"
     x ** 32
   end
   memoize :foo
end

Got it. I had used "include" rather than "extend". Your solution
applies memoization to all objects of a class. The original memoize
applied memoize on a per object basis.

I need to test and digest.

8

Overriding methods usually produces warnings. It's best to remove the
methods first.

I was wondering if warnings might be the motive. I am glad to get confirmation.

Actually, I think this is the cleanest solution. In theory you could
also offer a custom append_features() which defines the methods on the
class or module including Memoize itself. That's more complex and not as
easy to understand, though, in my opinion.

Cool. I had been thinking I wanted to use your version of Memoize in
conjunction with Rails' ActiveRecord::Base but ActiveRecord::Base
requires one to 'extend' it to use it. However it occurred to me that
since ActiveRecord::Base does not extend anything I could just extend
it instead of my actual class. It seems to work.

class ActiveRecord::Base
  extend Memoize
end

class Foo < ActiveRecord::Base
  def calc
    ...
  end
  memoize :calc
end

--Brian Buckley

9

Brian Buckley wrote:

class Foo
  extend Memoize

  def foo(x)
    puts "Calculating #{x} ** 32"
    x ** 32
  end
  memoize :foo
end

Got it. I had used "include" rather than "extend". Your solution
applies memoization to all objects of a class. The original memoize
applied memoize on a per object basis.

I need to test and digest.

Would folks like both options? memoize_all or something? Or just leave it alone?

- Dan

10

Brian Buckley wrote:

Cool. I had been thinking I wanted to use your version of Memoize in
conjunction with Rails' ActiveRecord::Base but ActiveRecord::Base
requires one to 'extend' it to use it. However it occurred to me that
since ActiveRecord::Base does not extend anything I could just extend
it instead of my actual class. It seems to work.

class ActiveRecord::Base
  extend Memoize
end

class Foo < ActiveRecord::Base
  def calc
    ...
  end
  memoize :calc
end

Please note that extend is not the same thing as inheriting from another class. You can inherit from ActiveRecord::Base and still have a extend(Memoize).

11

With a slight adjustment, Florian's version can be used at the top
level too (at the cost of the instance variable @cache in main):

module Memoize
  def memoize(name)
    name = name.to_sym
    if self.to_s == "main"
      old_method = class<<self;self;end.instance_method(name)
      klass = Object
    else
      old_method = instance_method(name)
      klass = self
    end
    klass.send(:remove_method, name)
    klass.send(:define_method, name) do |*args|
      @cache ||= {}
      signature = [name] + args

      if @cache.include?(signature) then
        @cache[signature]
      else
        @cache[signature] = old_method.bind(self).call(*args)
      end
    end
  end
end

class D
  extend Memoize
  attr_accessor :value
  def initialize(value)
    @value = value
  end
  def t_memo(*args)
    puts "calculating #{args.inspect}"
    args.map { |x| x * value }
  end
  memoize(:t_memo)

end

d = D.new(10)
p d.t_memo(7,8)
p d.t_memo(7,8)

p d.t_memo(1,2)
p d.t_memo(1,2)

e = D.new(20)
p e.t_memo(1,2)
p e.t_memo(7,8)
p d.t_memo(7,8)

p d.instance_eval { @cache }

p e.instance_eval { @cache }

extend Memoize

def t_memo(*args)
  puts "calculating #{args.inspect}"
  args.map { |x| x * 5 }
end

memoize :t_memo

p t_memo(1,2,3)
p t_memo(1,2,3)
p t_memo(7,8)

p @cache

__END__
calculating [7, 8]
[70, 80]
[70, 80]
calculating [1, 2]
[10, 20]
[10, 20]
calculating [1, 2]
[20, 40]
calculating [7, 8]
[140, 160]
[70, 80]
{[:t_memo, 1, 2]=>[10, 20], [:t_memo, 7, 8]=>[70, 80]}
{[:t_memo, 1, 2]=>[20, 40], [:t_memo, 7, 8]=>[140, 160]}
calculating [1, 2, 3]
[5, 10, 15]
[5, 10, 15]
calculating [7, 8]
[35, 40]
{[:t_memo, 7, 8]=>[35, 40], [:t_memo, 1, 2, 3]=>[5, 10, 15]}

Regards,

Sean

···

On 11/3/05, Daniel Berger <Daniel.Berger@qwest.com> wrote:

Would folks like both options? memoize_all or something? Or just leave it alone?