Multiple asserts in a single unit test

ruby-talk
1

Hi all,

I came across an article regarding unit testing philosophy that I
thought was interesting:

http://weblogs.asp.net/rosherove/archive/2005/04/14/AvoidMultipleAsserts.aspx

This sparked a conversation on IRC regarding coming up with a way to
have asserts continue even if one of them fails within a unit test. Is
there a way to alter the behavior of test-unit so that it doesn't bail
out on subsequent asserts if one fails? Something builtin to test-unit
or a simple hack?

Any opinions on multiple asserts within a single unit test? Any
opinions on what the default behavior should be for test-unit with
regards to multiple asserts where one fails?

Regards,

Dan

2

Daniel Berger schrieb:

I came across an article regarding unit testing philosophy that I
thought was interesting:

ISerializable - Roy Osherove's Blog - Try to avoid multiple asserts in a single unit test

This sparked a conversation on IRC regarding coming up with a way to
have asserts continue even if one of them fails within a unit test. Is
there a way to alter the behavior of test-unit so that it doesn't bail
out on subsequent asserts if one fails? Something builtin to test-unit
or a simple hack?

It may not be simple, but at least it's a hack:

   require 'test/unit'

   module ErrorCollector

     def collecting_errors
       is_collecting = @is_collecting
       @is_collecting = true
       yield
     ensure
       @is_collecting = is_collecting
     end

     def raise( * )
       super
     rescue Test::Unit::AssertionFailedError
       handle_error( :add_failure, $! )
     rescue StandardError, ScriptError
       handle_error( :add_error, $! )
     end

     def handle_error( method, error )
       bck = error.backtrace
       bck.shift
       if @is_collecting
         bck.slice!( 5, 2 )
         send( method, error.message, bck )
       else
         Kernel.raise( error, error.message, error.backtrace )
       end
     end

   end

   class SampleTest < Test::Unit::TestCase

     include ErrorCollector

     def test_multiple_errors
       collecting_errors do
         assert_equal( 1, 2 )
         assert_equal( 2, 3 )
       end
       assert_equal( 3, 4 )
       assert_equal( 4, 5 )
     end

   end

This is the output:

   Loaded suite C:/tmp/r
   Started
   FFF
   Finished in 0.015 seconds.

     1) Failure:
   test_multiple_errors(SampleTest) [C:/tmp/r.rb:42]:
   <1> expected but was
   <2>.

     2) Failure:
   test_multiple_errors(SampleTest) [C:/tmp/r.rb:43]:
   <2> expected but was
   <3>.

     3) Failure:
   test_multiple_errors(SampleTest) [C:/tmp/r.rb:45]:
   <3> expected but was
   <4>.

   1 tests, 3 assertions, 3 failures, 0 errors

Regards,
Pit

3

Daniel Berger wrote:

Any opinions on multiple asserts within a single unit test?

The distinction is roughly between testing greenfield code with testing
legacy systems.

In greenfield code, test-first can force so much decoupling that you don't
/want/ to assert more than one thing in each case. So setting "one assertion
per case" as a design goal

Some tests stretch the definition of "one assertion":

  def test_def_paren
    tokenize( "def foo(bar)" )
    assert_next_token :keyword, "def "
    assert_next_token :method, "foo"
    assert_next_token :punct, "("
    assert_next_token :ident, "bar"
    assert_next_token :punct, ")"
  end

That is an assertion of one variable (a hidden @member), chained out to show
each tested detail in the variable. (It's from Jamis Buck's Syntax module.)

Legacy systems, by contrast, typically test at a higher level. If you test a
Tk GUI, simulating a click on a Tk Canvas might change many configurations.
Rather than waste the test state, and the time required to paint a canvas,
write a list of assertions that check each important configuration.

Any
opinions on what the default behavior should be for test-unit with
regards to multiple asserts where one fails?

If you have a debugger, such as for VC++, an assertion should raise a
breakpoint, giving a programmer the option to run more lines.

Unattended, some assertions should drop-dead, and some should keep going.
However, assertion systems already have numerous permutations (assert_equal,
assert_match, assert_nil, modulo _not, etc.). Adding another permutation
would blow our minds.

In C++, I use a raw ASSERT() to drop an entire test run dead, and use
CHECK() for relatively recoverable situations. CHECK(), in unattended mode,
keeps going.

Ultimately, the rule for Test-Driven Development is you don't coddle the
test failure. Your rig should provide automated navigation to the failing
assertion, and should reflect its variables and values. You should either
instantly fix the problem or instantly use Undo to run back to the last
failing state. Coddling the test failures - making a list of them, logging
them, e-mailing them to your boss - are all secondary considerations.

So, all assertions could just drop a test run dead, and TDD will work fine.
Your nightly unattended test run will always pass, so recovering from
assertions is irrelevant.

···

--
  Phlip
  http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces

4

IMO, the number of passes plus the number of fails should equal the
number of asserts, regardless of which tests fail. Phlip seems to
agree, but that there should only be one assertatin per test anyway,
so that this iregularity won't be a problem. I find that an artificial
constraint on writing unit tests, so imo tests should continue even
after a failed test.

Also, take this test:

def test_too_true
  assert false, "One failed"
end

def test_true
  assert false, "Two failed"
  assert false, "Three failed"
end

With "break on first fail", this gives:

"One failed"
"Two failed"

Fixing "One" gives:

"Two failed"

Then you fix "Two" and it gets replaced with:

"Three failed"

In the first case, fixing a fail removed it from the list. In the
second case, it was simply replaced with something else, and the only
difference was which test the assertation was put in. It shouldn't
matter where the asserts go, the behaviour of solving something should
be that you stop seeing that it fails, nothing more need happen.

Douglas

···

On 4/15/05, Daniel Berger <djberg96@hotmail.com> wrote:

This sparked a conversation on IRC regarding coming up with a way to
have asserts continue even if one of them fails within a unit test. Is
there a way to alter the behavior of test-unit so that it doesn't bail
out on subsequent asserts if one fails?

5

I prefer having multiple tests per method, each testing some other path through the method. This seems to be a workable alternative (for me!) to continuing despite failures.

I feel I get the same amount of information regarding potential failure points that the continue method gives. It also encourages me to write shorter tests because continuing could lead to tests that are too long.

PGP.sig (194 Bytes)

···

On 14 Apr 2005, at 21:09, Daniel Berger wrote:

Hi all,

I came across an article regarding unit testing philosophy that I
thought was interesting:

ISerializable - Roy Osherove's Blog - Try to avoid multiple asserts in a single unit test

This sparked a conversation on IRC regarding coming up with a way to
have asserts continue even if one of them fails within a unit test. Is
there a way to alter the behavior of test-unit so that it doesn't bail
out on subsequent asserts if one fails? Something builtin to test-unit
or a simple hack?

Any opinions on multiple asserts within a single unit test? Any
opinions on what the default behavior should be for test-unit with
regards to multiple asserts where one fails?

--
Eric Hodel - drbrain@segment7.net - http://segment7.net
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

6

Quote of the week :slight_smile:

martin

···

Pit Capitain <pit@capitain.de> wrote:

> out on subsequent asserts if one fails? Something builtin to test-unit
> or a simple hack?

It may not be simple, but at least it's a hack:

7

In greenfield code, test-first can force so much decoupling that you don't
/want/ to assert more than one thing in each case. So setting "one

assertion

per case" as a design goal

sigh<

....tends to decouple.

Friggin' Alzheimers...

···

--
  Phlip
  http://industrialxp.org/community/bin/view/Main/TestFirstUserInterfaces