Iterating in a test

Paul_Brannan · 10 January 2003 16:29

Short question: if I have a test that iterates over a list of items, how
can I get assertion failure messages and exceptions to report which
iteraation I am on?

Longer question:

Suppose I have a test like this:

require ‘test/unit’

def f(x)
return x**2
end

class MyTest < Test::Unit::TestCase
def initialize(*args)
super(*args)
p *args
end

def test_x_squared
  assert_equal 1, f(1)
  assert_equal 4, f(2)
  assert_equal 9, f(3)
end

end

I can rewrite test_x_squared like this:

def test_x_squared
  for i in 1..3 do
    assert_equal i*i, f(i)
  end
end

but if the test fails (I rewrite f(x) to return x**4), I get:

"test_x_squared"
Loaded suite test
Started…
…
Failure occurred in test_x_squared(MyTest) [test.rb:21]: Expected <4> but was <16>

which isn’t very helpful. If f(x) raises an exception, the message is
also not very helpful:

"test_x_squared"
Loaded suite test
Started…
…
Error occurred in test_x_squared(MyTest): RuntimeError: f(x) failed
test.rb:4:in f' test.rb:22:intest_x_squared’
test.rb:21:in each' test.rb:21:intest_x_squared’
test.rb:20

The solution I came up with is something like this:

def test_x_squared
  for i in 1..3 do
    appendage = "while testing for i=#{i}"
    begin
      assert_equal i*i, f(i), appendage
    ensure
      $!.message.replace($!.message + ' ' + appendage) if $!
    end
  end
end

but it seems like there should be a cleaner or more generic solution
than this. What is it?

Thanks,

Paul

Martin_Hart · 10 January 2003 16:38

On 1/10/03, 4:29:31 PM, Paul Brannan pbrannan@atdesk.com wrote regarding
iterating in a test:

Short question: if I have a test that iterates over a list of items, how
can I get assertion failure messages and exceptions to report which
iteraation I am on?

I believe that each assert function can take a message string which is
dumped if the test fails:

10.times { |i|
assert_equal(1, f(i), “Testing f where i=#{i}”)
}

Cheers,

Martin

Paul_Brannan · 10 January 2003 16:51

This alone does not solve the whole problem; f(i) could raise an
exception, and if it does I do not know what i is.

Paul

···

On Sat, Jan 11, 2003 at 01:38:09AM +0900, Martin Hart wrote:

I believe that each assert function can take a message string which is
dumped if the test fails:

10.times { |i|
assert_equal(1, f(i), “Testing f where i=#{i}”)
}

Nathaniel_Talbott · 10 January 2003 16:58

Paul Brannan said:

This alone does not solve the whole problem; f(i) could raise an
exception, and if it does I do not know what i is.

How about (untested):

10.times { |i|
assert_nothing_raised(“Testing f where i=#{i}”) {
assert_equal(1, f(i), “Testing f where i=#{i}”)
}
}

Nathaniel

Martin_Hart · 10 January 2003 17:05

On 1/10/03, 4:51:02 PM, Paul Brannan pbrannan@atdesk.com wrote regarding
Re: iterating in a test:

This alone does not solve the whole problem; f(i) could raise an
exception, and if it does I do not know what i is.

Ok, this is messy, ugly, and probably not what you want

[martin@dorfl martin]$ cat test.rb

require ‘test/unit’

def f(i)
raise “anException” if i==1
i
end

class TestX < Test::Unit::TestCase

    def testF
      10.times { |i|
                    assert_nothing_raised("i was #{i}") {
                            assert_equal(0, f(i), "i was #{i}")
                    }
            }
    end

end

As you can see, the problem now is that if the assert_equal fails, we
actually get an “Exception raised” error. However, you do get to see what
i is

Cheers,
Martin

Gavin_Sinclair · 11 January 2003 00:17

My lateral answer to this is use the debugger. If an exception is
thrown, run the test in the debugger, catch the exception, wander the
stack frames and explore.

That’s a messy “solution”, but to me is justified because I rarely
perform such iterative testing. Also, most of the time it will
succeed, so the message is rarely necessary. Your milage probably
varies.

Gavin

···

On Saturday, January 11, 2003, 3:51:02 AM, Paul wrote:

On Sat, Jan 11, 2003 at 01:38:09AM +0900, Martin Hart wrote:

I believe that each assert function can take a message string which is
dumped if the test fails:

10.times { |i|
assert_equal(1, f(i), “Testing f where i=#{i}”)
}

This alone does not solve the whole problem; f(i) could raise an
exception, and if it does I do not know what i is.

Paul_Brannan · 10 January 2003 18:07

In this case I still have to duplicate the message for the
assert_nothing_raised and the assert_equal call.

What I think I’d like to see is something like this:

def test_foo
10.times do |i|
performing_action(“Testing f(x) where i=#{i}”) do
assert_equal 1, f(i)
some_helper_function(1)
end
end
end

def some_helper_function(i)
performing_action(“inside the helper function”) do
# make some assertions here
end
end

The performing_action() method could append the passed string onto an
array at the start of the block (and remove it at the end), so I have a
list of actions that I’m in the middle of performing. This list of
actions should be printed for every failed assertion and every exception
that escapes.

I’m not sure where to begin implementing this, though, or whether it’s
possible to implement it without modifying Test::Unit.

Paul

···

On Sat, Jan 11, 2003 at 01:58:07AM +0900, Nathaniel Talbott wrote:

How about (untested):

10.times { |i|
assert_nothing_raised(“Testing f where i=#{i}”) {
assert_equal(1, f(i), “Testing f where i=#{i}”)
}
}

Paul_Brannan · 13 January 2003 14:39

The problem with using the debugger is that it may not be possible to
reproduce the bug in the debugger. This is particularly true when I’ve
run the test 1000 times, and the 1000th time the test fails due to a
race condition. In that case, I want the test to give me as much
information as possible so I can track down the problem without
re-running the test.

Paul

···

On Sat, Jan 11, 2003 at 09:17:58AM +0900, Gavin Sinclair wrote:

My lateral answer to this is use the debugger. If an exception is
thrown, run the test in the debugger, catch the exception, wander the
stack frames and explore.

Nathaniel_Talbott · 22 January 2003 06:31

How about (untested):

10.times { |i|
assert_nothing_raised(“Testing f where i=#{i}”) {
assert_equal(1, f(i), “Testing f where i=#{i}”)
}
}

In this case I still have to duplicate the message for the
assert_nothing_raised and the assert_equal call.

Hmmm… you’re right - I should’ve used a temp var to store the message
;-). Seriously, that doesn’t seem like a hardship to me, and it seems
much simpler than what you propose below…

What I think I’d like to see is something like this:

def test_foo
10.times do |i|
performing_action(“Testing f(x) where i=#{i}”) do
assert_equal 1, f(i)
some_helper_function(1)
end
end
end

def some_helper_function(i)
performing_action(“inside the helper function”) do
# make some assertions here
end
end

The performing_action() method could append the passed string onto an
array at the start of the block (and remove it at the end), so I have
a list of actions that I’m in the middle of performing. This list of
actions should be printed for every failed assertion and every
exception that escapes.

def performing_action(message) # Not tested
@message_stack ||=
@message_stack.push(message)
begin
yield
rescue Exception
raise $!, [$!.message, *@message_stack].join(“\n”), $!.backtrace
ensure
@message_stack.pop
end
end

It’s seems overly complex for me to want to use it, though; if the
simpler solution I gave above wouldn’t work, I’d probably start looking
to refactor my tests. Just my $0.02.

HTH,

Nathaniel

<:((><

···

Paul Brannan [mailto:pbrannan@atdesk.com] wrote:

On Sat, Jan 11, 2003 at 01:58:07AM +0900, Nathaniel Talbott wrote:
RoleModel Software, Inc.
EQUIP VI

Paul_Brannan · 22 January 2003 21:32

def performing_action(message) # Not tested
@message_stack ||=
@message_stack.push(message)
begin
yield
rescue Exception
raise $!, [$!.message, *@message_stack].join(“\n”), $!.backtrace
ensure
@message_stack.pop
end
end

This works well in Ruby 1.7/1.8. In Ruby 1.6 this doesn’t work as well,
though:

class Exc < Exception
def initialize(a, b, c)
super(“Exception: #{a}, #{b}, #{c}”)
end
end

performing_action(‘playing with Ruby’) do
performing_action(‘doing some silly things’) do
raise Exc.new(1, 2, 3)
end
end

[pbrannan@zaphod tmp]$ ruby -v
ruby 1.6.8 (2002-12-24) [i686-linux]
[pbrannan@zaphod tmp]$ ruby test.rb
test.rb:7:in initialize': wrong # of arguments(1 for 3) (ArgumentError) playing with Ruby from test.rb:7:in exception’
from test.rb:7:in raise' from test.rb:7:in performing_action’
from test.rb:32
from test.rb:31:in `performing_action’
from test.rb:31

So instead I have:

def performing_action(action)
begin
yield
ensure
$!.message.replace($!.message + "\n\twhile " + action) if $!
end
end

[pbrannan@zaphod tmp]$ ruby test.rb
test.rb:32: Exception: 1, 2, 3 (Exc)
while doing some silly things
while playing with Ruby
from test.rb:31:in performing_action' from test.rb:31 from test.rb:30:in performing_action’
from test.rb:30

Or alternately:

module ExceptionActions
attr_reader :actions

def self.extend_object(obj)
  super(obj)
  obj.instance_eval { @actions = [] }
end

def to_s
  return super + @actions.map { |a| "\n\twhile #{a}" }.join
end

alias_method :message, :to_s
alias_method :to_str, :to_s

end

def performing_action(action)
begin
yield
ensure
if $! then
if not ExceptionActions === exc then
$!.extend(ExceptionActions)
end
$!.actions.push(action)
end
end
end

The latter solution is a little more verbose but also a little more
robust (it still works even if $!.message is not a string).

It’s seems overly complex for me to want to use it, though; if the
simpler solution I gave above wouldn’t work, I’d probably start looking
to refactor my tests. Just my $0.02.

Refactoring my tests is what brought this about. I had many tests doing
very similar things, but varying by only a few parameters. My solution
was to consolidate those tests into a few small tests. Perhaps there’s
a better solution that I’m not aware of.

Paul

···

On Wed, Jan 22, 2003 at 03:31:01PM +0900, nathaniel@NOSPAMtalbott.ws wrote:

Topic		Replies	Views
Comparing strings and integers in Test::Unit ruby-talk	2	109	26 February 2003
Test::Unit Newbie Question regarding loops ruby-talk	12	134	23 February 2010
Test::Unit need help with assert ruby-talk	4	114	27 October 2006
Test::Unit: assert in loop? ruby-talk	2	109	19 June 2003
Assertions in Ruby ruby-talk	0	67	4 October 2002

Iterating in a test

Related topics