Benchmark in MiniTest

ruby-talk
1

  def test_hot_is_constant
    assert_performance_constant 0.9999 do
      @reddit.hot
    end
  end

  def bench_my_algorithm
    assert_performance_linear 0.9999 do |n| # n is a range value
      n.times do
        @reddit.hot
      end
    end
  end

the output i get is:
TestReddit 1 10 100 1000 10000
bench_hot 0.000039 0.000052 0.000032 0.000033 0.000032
bench_hot2 0.000046 0.000047 0.000088 0.000596 0.005650

my question - Is there a way to set the maximum time for a method to run?
if it goes above it i want it to fail.
what does assert_performance_constant mean? is it a constant speed in relation to the argument the method accepts?

Thanks

2

I guess this works (ruby 1.9.2)

  def test_speed
   Timeout.timeout(0.1) {@reddit.hot}
  end

but I am still curious about the different benchmark methods minitest provides.

3

gist:813736 · GitHub

def test_hot_is_constant
   assert_performance_constant 0.9999 do
     @reddit.hot
   end
end

def bench_my_algorithm
   assert_performance_linear 0.9999 do |n| # n is a range value
     n.times do
       @reddit.hot
     end
   end
end

the output i get is:
TestReddit 1 10 100 1000 10000
bench_hot 0.000039 0.000052 0.000032 0.000033 0.000032
bench_hot2 0.000046 0.000047 0.000088 0.000596 0.005650

my question - Is there a way to set the maximum time for a method to run?
if it goes above it i want it to fail.

No. I'm not sure it makes too much sense. On a slower or more loaded machine, you'd get false negatives.

what does assert_performance_constant mean? is it a constant speed in relation to the argument the method accepts?

8511 % ri MiniTest::Unit::TestCase.assert_performance_constant
= MiniTest::Unit::TestCase.assert_performance_constant

(from gem minitest-2.0.2)

···

On Feb 6, 2011, at 16:15 , Oren wrote:
------------------------------------------------------------------------------
  assert_performance_constant(threshold = 0.99, &work)

------------------------------------------------------------------------------

Runs the given work and asserts that the times gathered fit to match a
constant rate (eg, linear slope == 0) within a given error threshold.

Fit is calculated by #fit_constant.

Ranges are specified by ::bench_range.

Eg:

  def bench_algorithm
    assert_performance_constant 0.9999 do |x|
      @obj.algorithm
    end
  end

4

Please be aware your are testing if Integer#times is liner in your
second test, not @reddit.hot.
(which does not take any parameter from your code, and is then very
likely to have constant time).

A good example would be:

def fib(n)
  # my super code to compute the n^th Fibonacci term in linear time
end

assert_performance_linear 0.9999 do |n|
  fib(n)
end

The only good usage of Integer#times (except for testing #times
itself) in assert_performance_*, is to increase the time when the
block runs too fast (like all n <= 10 000 runs in less than 0.001 s)
You then want to use a constant to call #times on it (like 1000.times
{ fib(n) }), *never* n.
You could also use the :bench_range option, probably better.

Regards,
Benoit Daloze

···

On 7 February 2011 01:15, Oren <orengolan@gmail.com> wrote:

gist:813736 · GitHub

def test_hot_is_constant
assert_performance_constant 0.9999 do
@reddit.hot
end
end

def bench_my_algorithm
assert_performance_linear 0.9999 do |n| # n is a range value
n.times do
@reddit.hot
end
end
end

the output i get is:
TestReddit 1 10 100 1000 10000
bench_hot 0.000039 0.000052 0.000032 0.000033 0.000032
bench_hot2 0.000046 0.000047 0.000088 0.000596 0.005650

my question - Is there a way to set the maximum time for a method to run?
if it goes above it i want it to fail.
what does assert_performance_constant mean? is it a constant speed in relation to the argument the method accepts?

Thanks

5

ri is your friend:

8512 % ri MiniTest::Unit::TestCase | grep assert_performance
  assert_performance
  assert_performance_constant
  assert_performance_exponential
  assert_performance_linear
  assert_performance_power

···

On Feb 6, 2011, at 16:30 , Oren wrote:

I guess this works (ruby 1.9.2)

def test_speed
  Timeout.timeout(0.1) {@reddit.hot}
end

but I am still curious about the different benchmark methods minitest provides.

6

Thanks Ryan!