Return values from assertions

Eivind Eklund and I have been discussing whether assertions ought to
have return values in Test::Unit, but while he’s quite convinced that
they shouldn’t (I’ll let him summarize his own viewpoint), I’m having
trouble making up my mind. Thus I’m coming to the community looking for
three kinds of input, though you’re certainly welcome to throw in
whatever spare change you have lying around:

  1. Thoughts on the overall idea of assertions returning values,
    particularly pitfalls you see with it.

  2. Code examples using the current return values from #assert_match and
    #assert_raises.

  3. Code examples showing the use of return values from other assertions.

The code examples are particularly important, as I want to evaluate what
returning values from assertions does to the test code.

Thanks!

Nathaniel

<:((><

···

RoleModel Software, Inc.
EQUIP VI

nathaniel@NOSPAMtalbott.ws writes:

Eivind Eklund and I have been discussing whether assertions ought to
have return values in Test::Unit, but while he’s quite convinced that
they shouldn’t (I’ll let him summarize his own viewpoint), I’m having
trouble making up my mind. Thus I’m coming to the community looking for
three kinds of input, though you’re certainly welcome to throw in
whatever spare change you have lying around:

  1. Thoughts on the overall idea of assertions returning values,
    particularly pitfalls you see with it.

No thoughts here, though I find having assert_raises return the
exception raised useful for further testing on the exception object
(below). Similar arguments for assert_match, though I had not thought
to do that until now.

  1. Code examples using the current return values from #assert_match and
    #assert_raises.

This is using RUnit’s assert_exception, but you get the idea. I use
it all the time to make sure the expected exception is the one I think
it is by examining the message.

defer_reason = "I might not like you any more"
lda, log = new_lda
e = assert_exception(RFilter::DeliveryAgent::DeliveryDefer) {
  lda.defer(defer_reason)
}
assert_equal(e.message, defer_reason)
  1. Code examples showing the use of return values from other assertions.

I don’t know if Test::Unit has the equivalent of RUnit’s
assert_no_exception, but I have used it this way:

h = RMail::Header.new
# This time is out of the range that can be represented by a
# Time object.
h.add_raw("Date: Sun, 14 Jun 2065 05:51:55 +0200")
t = assert_no_exception {
  h.date
}
assert_nil(t)
  1. Thoughts on the overall idea of assertions returning values,
    particularly pitfalls you see with it.

No thoughts here, though I find having assert_raises return
the exception raised useful for further testing on the
exception object (below). Similar arguments for
assert_match, though I had not thought to do that until now.

First of all, thanks for replying, Matt! I’m sorry it has taken so long
for me to get back to this.

In response to your statement about finding the exception return value
useful, I guess the question is, is it more readable (and not too
difficult) to do one of the following instead:

begin
raise “exception message”
flunk(“Should have thrown an exception”)
rescue Exception => e
assert_equal(“exception message”, e.message, “Expected a message”)
end

or,

assert_raises_with_message(Exception, “exception message”, “Expected
exception”) do
raise “exception message”
end

I realize the name in the latter example is perhaps not the best, but
it’s the idea of using a more specific assertion for the case of wanting
to check the message. Of course, there are other potentially interesting
bits of an exception, which is what I personally don’t like about it
(and I don’t like the verbosity of the first way).

  1. Code examples using the current return values from #assert_match
    and #assert_raises.

This is using RUnit’s assert_exception, but you get the idea.
I use it all the time to make sure the expected exception is
the one I think it is by examining the message.

defer_reason = "I might not like you any more"
lda, log = new_lda
e = assert_exception(RFilter::DeliveryAgent::DeliveryDefer) {
  lda.defer(defer_reason)
}
assert_equal(e.message, defer_reason)

Let me rewrite this with my two alternatives above:

defer_reason = “I might not like you any more”
lda, log = new_lda
begin
lda.defer(defer_reason)
flunk
rescue RFilter::DeliveryAgent::DeliveryDefer => e
assert_equal(e.message, defer_reason)
end

or

defer_reason = “I might not like you any more”
lda, log = new_lda
assert_exception(RFilter::DeliveryAgent::DeliveryDefer, defer_reason)
{
lda.defer(defer_reason)
}

Hmmm… after seeing that, I’m leaning towards leaving the return value
to #assert_raises. The first way is just too much work, and the second
way doesn’t give me enough power. It just seems more robust to allow the
ability to inspect anything in the exception, instead of being
constrained by what the framework designer thought you should be able to
check.

  1. Code examples showing the use of return values from other
    assertions.

I don’t know if Test::Unit has the equivalent of RUnit’s
assert_no_exception, but I have used it this way:

h = RMail::Header.new
# This time is out of the range that can be represented by a
# Time object.
h.add_raw("Date: Sun, 14 Jun 2065 05:51:55 +0200")
t = assert_no_exception {
  h.date
}
assert_nil(t)

I don’t really understand what you’re testing here…
#assert_no_exception (or, in Test::Unit’s case, #assert_nothing_raised)
will fail already if an exception is thrown, so why the extra check
afterwards? From looking at RubyUnit’s source, there doesn’t even seem
to be an explicit return value from this method.

What I’m thinking at the moment is:

  1. Leave in the return value from #assert_raises.
  2. Remove the return value from #assert_match.
  3. In general, avoid returning (documented) values from assertions.

To clarify #3, I don’t plan on returning nil from all the assertions,
but I do plan on not explicitly returning something useful from them,
and saying that if you depend on something that is returned as a side
effect, too bad for you if it changes.

Thoughts?

Nathaniel

<:((><

···

Matt Armstrong [mailto:matt@lickey.com] wrote:

RoleModel Software, Inc.
EQUIP VI

nathaniel@NOSPAMtalbott.ws writes:

Hmmm… after seeing that, I’m leaning towards leaving the return
value to #assert_raises. The first way is just too much work, and
the second way doesn’t give me enough power.

Agreed. The first way doesn’t immediately convey the intent of the
test either – i.e. that we’re primarily checking for exception stuff.

  1. Code examples showing the use of return values from other
    assertions.

I don’t know if Test::Unit has the equivalent of RUnit’s
assert_no_exception, but I have used it this way:

h = RMail::Header.new
# This time is out of the range that can be represented by a
# Time object.
h.add_raw("Date: Sun, 14 Jun 2065 05:51:55 +0200")
t = assert_no_exception {
  h.date
}
assert_nil(t)

I don’t really understand what you’re testing here…

User error here. The assert_nil is clearly meaningless and works by
accident.

I’ve often wondered how useful assert_no_exception actually is, given
that the framework reports uncaught exceptions anyway.

What I’m thinking at the moment is:

  1. Leave in the return value from #assert_raises.
  2. Remove the return value from #assert_match.
  3. In general, avoid returning (documented) values from assertions.

To clarify #3, I don’t plan on returning nil from all the
assertions, but I do plan on not explicitly returning something
useful from them, and saying that if you depend on something that is
returned as a side effect, too bad for you if it changes.

Given that I’ve only ever meaningfully used the return value of
assert_raises, that sounds good to me. #3 is not my style (I’d just
return nil everywhere) but it is hard to come up with any real reason
to go one way or the other.

I’ve often wondered how useful assert_no_exception actually
is, given that the framework reports uncaught exceptions anyway.

The only advantage I see is if you wanted a particular exception that
could be thrown to show up as a failure instead of an error. This can be
particularly useful if you want to add an assertion message if/when that
exception is thrown.

What I’m thinking at the moment is:

  1. Leave in the return value from #assert_raises.
  2. Remove the return value from #assert_match.
  3. In general, avoid returning (documented) values from assertions.

To clarify #3, I don’t plan on returning nil from all the
assertions,
but I do plan on not explicitly returning something useful from
them,
and saying that if you depend on something that is returned as a
side
effect, too bad for you if it changes.

Given that I’ve only ever meaningfully used the return value
of assert_raises, that sounds good to me. #3 is not my style
(I’d just return nil everywhere) but it is hard to come up
with any real reason to go one way or the other.

Is there a good rule of thumb for what methods should do
when they don’t have a useful value to return? I can’t remember seeing
any code that was careful to always return nil, but that could just be
my limited reading.

Nathaniel

<:((><

···

Matt Armstrong [mailto:matt@lickey.com] wrote:

RoleModel Software, Inc.
EQUIP VI

I don’t know about rule of thumb, but I would prefer to see nil
returned. We rely on documentation to specify APIs in Ruby, since no
type declarations do that for us, and keeping things uniform and
documented would serve Test::Unit well.

Gavin

···

On Thursday, February 6, 2003, 2:42:16 PM, nathaniel wrote:

Is there a good rule of thumb for what methods should do
when they don’t have a useful value to return? I can’t remember seeing
any code that was careful to always return nil, but that could just be
my limited reading.

nathaniel@NOSPAMtalbott.ws writes:

···

Matt Armstrong [mailto:matt@lickey.com] wrote:

I’ve often wondered how useful assert_no_exception actually
is, given that the framework reports uncaught exceptions anyway.

The only advantage I see is if you wanted a particular exception
that could be thrown to show up as a failure instead of an
error. This can be particularly useful if you want to add an
assertion message if/when that exception is thrown.

That hits the crux of my question – the only practical difference
between a failure and an error is that the error can have a nice error
message?

That hits the crux of my question – the only practical difference
between a failure and an error is that the error can have a nice error
message?

Doesn’t an unhandled exception stop execution of the entire test suite,
whereas a failure only halts that particular test?

Chris

The conceptual difference is practical enough for me.

Conceptual difference:

  • failure: specifically tested condition that, well, failed
  • error: unexpected, uncaught exception

If you expect that certain code will throw an exception, you test for
it and record a success or failure. If you don’t expect an exception,
and you get one, then either your code or your test is seriously
wrong, in a different kind of way to a failed assertion.

Therefore, failures and errors highlight different problems in the
code, and this is a practical difference.

You probably knew all that and were angling at something else…

Gavin

···

On Friday, February 7, 2003, 2:52:38 AM, Matt wrote:

nathaniel@NOSPAMtalbott.ws writes:

Matt Armstrong [mailto:matt@lickey.com] wrote:

I’ve often wondered how useful assert_no_exception actually
is, given that the framework reports uncaught exceptions anyway.

The only advantage I see is if you wanted a particular exception
that could be thrown to show up as a failure instead of an
error. This can be particularly useful if you want to add an
assertion message if/when that exception is thrown.

That hits the crux of my question – the only practical difference
between a failure and an error is that the error can have a nice error
message?

No, exceptions just stop the current test case, just as failures do.

···

On Thu, 2003-02-06 at 11:56, Chris Morris wrote:

Doesn’t an unhandled exception stop execution of the entire test suite,
whereas a failure only halts that particular test?


– Jim Weirich jweirich@one.net http://w3.one.net/~jweirich

“Beware of bugs in the above code; I have only proved it correct,
not tried it.” – Donald Knuth (in a memo to Peter van Emde Boas)

Gavin Sinclair gsinclair@soyabean.com.au writes:

···

On Friday, February 7, 2003, 2:52:38 AM, Matt wrote:

Matt Armstrong [mailto:matt@lickey.com] wrote:

That hits the crux of my question – the only practical difference
between a failure and an error is that the error can have a nice error
message?

The conceptual difference is practical enough for me.

Conceptual difference:

  • failure: specifically tested condition that, well, failed
  • error: unexpected, uncaught exception

If you expect that certain code will throw an exception, you test for
it and record a success or failure. If you don’t expect an exception,
and you get one, then either your code or your test is seriously
wrong, in a different kind of way to a failed assertion.

Therefore, failures and errors highlight different problems in the
code, and this is a practical difference.

Even more to the point, I don’t see a point to RubyUnit’s
assert_no_exception or Test::Unit’s assert_nothing_raised.
Conceptually, you’re saying “this code shouldn’t raise an exception
that is not handled” but that is pretty much universally true.


matt

I see a point. It enables you to unit-test an operation that doesn’t
return anything, but which raises an exception on failure. You could
just call it and let it be an error instead of a failure, but
assertions are a good way to document what you’re testing.

It would perhaps be better to specify which exceptions are “expected”,
because obviously you can have some sort of run-time exception that
you didn’t expect, and should be informed about. I’ve done this a few
times:

begin
method_to_be_tested
rescue ApplicationSpecificException
fail “…”
end

Gavin

···

On Friday, February 7, 2003, 2:01:41 PM, Matt wrote:

Therefore, failures and errors highlight different problems in the
code, and this is a practical difference.

Even more to the point, I don’t see a point to RubyUnit’s
assert_no_exception or Test::Unit’s assert_nothing_raised.
Conceptually, you’re saying “this code shouldn’t raise an exception
that is not handled” but that is pretty much universally true.