Test::Unit: assert_follows_spec() (or something like that)

ruby-talk
1

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

This can make unit tests shorter and more readable. We can transform the
spec into an HTML table, include it into the API documentation, etc.

The assert_follows_spec() could even perform several iterations of tests
if wanted, to test deterministic-ness.

···

–
dave

2

Good idea… for ages I have been using

io = [
[‘inputx’, ‘outputx’],
[‘bla’, ‘outputbla’],
[‘ruby’, ‘outputruby’],
]
input, expected = io.transpose
actual = input.map{|i| stuff(i) }
assert_equal(expected, actual)

If there are too many input/output entries, then its difficult to
quickly identify where the problem are located.

A more well-crafted assertion method, which are able to output
more precisely where the problems are located, would be greak.

···

David Garamond lists@zara.6.isreserved.com wrote:

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

This can make unit tests shorter and more readable. We can transform the
spec into an HTML table, include it into the API documentation, etc.

The assert_follows_spec() could even perform several iterations of tests
if wanted, to test deterministic-ness.

–
Simon Strandgaard

3

Hi Dave,

I just started to use Test::Unit (only to discover that it provides a framework for what I have done in a ad hoc way for years). Nice.

I assume that using assert_block() it is rather easy to implement assert_follows_spec().

However, the name “follows_spec” may be slightly misleading, because the true “spec” is more like the full suite of tests than the subset that is provided to assert_follows_spec(). Maybe “assert_calls” is a more appropriate name,
because what is done is that: calls with parameters and expected results.

The idea of running the calls multiple times is for sure useful sometimes.

About the generation of HTML documentation, I feel, again, that the full spec is not easy to grab. I would
rather like to see some RDoc tool that would consider test cases as “examples”. There would then be a need
for some cross-ref where you can jump to test cases that involve some method you’re looking at.

This way one would leverage the value of test cases: they would become more useful as documentation items.

I do agree that there is room for improvement regarding the size of tests. assert_calls() is one improvement.

Another one I can think of is a way to switch to the next test without creating a method for it. Today I tend to tests multiple things in a single test method, more like a sequence of method calls than individual methods. Coining a name for the test method is not easy (because it tests so many things). I end up calling them test_1, test_2 and so on, that is not very useful.

I would like something like “testing( msg)” where msg would complement the context that would be displayed if some assert bombs (in addition to the name of the test method as is done today). In the end the name of the test method should be more like the name of the “use case” I am testing. testing(msg) would describe the intermediary steps of
the “use case”.

I’ll dig further in Test::Unit source code if/when time permit.

Yours,

JeanHuguesRobert

···

At 15:31 29/05/2004 +0900, you wrote:

For deterministic functions/methods, the same set of inputs should always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

This can make unit tests shorter and more readable. We can transform the spec into an HTML table, include it into the API documentation, etc.

The assert_follows_spec() could even perform several iterations of tests if wanted, to test deterministic-ness.

–
dave

Web: @jhr is virteal, virtually real
Phone: +33 (0) 4 92 27 74 17

4

David Garamond wrote:

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

Of course the above should be written as :input, :output, etc. Too much
writing YAML, I guess :-).

Some more ideas:

  • max_delta for floats:

    {:input => [1, 3.0], :output => 0.333333333333333,
    :max_delta => 0.000000000000001}

  • support for non-deterministic func/methods, like ‘output_in’ (multiple
    possible outputs), ‘output_class’ and ‘output_range’ (for any acceptable
    output or a certain clas and inside a certain range). e.g. for testing
    Kernel#rand:

    [{:input => [1], :output => 0},
    {:input => [3], :output_in => [0, 1, 2]},
    {:input => , :output_class => Float,
    :output_range => 0…1.0}]

Basically, I’m seeing assert_follows_spec (or assert_calls, or whatever)
as simply an aggregate of lots of simpler asserts (assert_equals,
assert_raises, etc) as one big assert. The difference is, it’s shorter
(it saves some keystrokes), nicer/tidier, and the “spec” itself is a
data structure and can be changed/manipulated/displayed/etc more easily.

Of course, this “spec” in many cases cannot cover all aspects of the
testing, only simpler ones. So perhaps a better name for “spec” is desired.

···

–
dave

5

If you’ve got a spec like this, why not write a method to autogenerate
test methods?

···

David Garamond (lists@zara.6.isreserved.com) wrote:

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

This can make unit tests shorter and more readable. We can transform the
spec into an HTML table, include it into the API documentation, etc.

The assert_follows_spec() could even perform several iterations of tests
if wanted, to test deterministic-ness.

–
Eric Hodel - drbrain@segment7.net - http://segment7.net
All messages signed with fingerprint:
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

6

Jean-Hugues ROBERT wrote:

Hi

Moin.

About the generation of HTML documentation, I feel, again, that the
full spec is not easy to grab. I would rather like to see some RDoc
tool that would consider test cases as “examples”.

I’m currently embedding test cases into the documentation. They look a
lot like example code. I think this is commonly called “Specification by
example”. I’ve written a tool for automatically extracting and running
them. Here is an example:

 # Matches the space between words.
 # This is a zero-width anchor.
··· 

 #   re = Regexp::English.word_boundary
 #   text = "Hello World!"
 #   text.gsub(re, "*") # => "*Hello* *World*!"
 #   # Matches the word "and", but not "band"
 #   re2 = Regexp::English.new do
 #     literal("and").surrounded_by(word_boundary)
 #   end
 #   "band".match(re2)   # => nil
 #   "and".match(re2)[0] # => "and"
 def word_boundary; Node::Anchor.new(/\b/); end

If this test case where to fail you would get an useful error like this:

  1. Failure:
    test_word_boundary()
    [(eval):3:in eval' (eval):7:in test_word_boundary’
    (eval):3:in test_word_boundary']: c:/ruby/lib/ruby/1.8/test/unit/assertions.rb:32:in assert_block’<“"Hello World!"”> expected but was <“"Hello World!"”>.

Yours,
JeanHuguesRobert

Regards,
Florian Gross

7

David Garamond wrote:

David Garamond wrote:

For deterministic functions/methods, the same set of inputs should
always produces the same output. What about adding something like:

assert_follows_spec(method_or_class_name, spec)

Where spec is an array of input and output pairs (and optionally an
exception, if the input should not be accepted):

[{input => [1, 1, 1], output => 1},
{input => [8, 1, 2], output => 4},
{input => [8, 3, 4], output => 6},
{input => [4, 3, 2], output => 6},
{input => [1, 2, nil], exception => ArgumentError},
{input => [1, 2, 0], exception => ZeroDivisionError}]

Of course the above should be written as :input, :output, etc. Too much
writing YAML, I guess :-).

Some more ideas:

  • max_delta for floats:

    {:input => [1, 3.0], :output => 0.333333333333333,
    :max_delta => 0.000000000000001}

  • support for non-deterministic func/methods, like ‘output_in’ (multiple
    possible outputs), ‘output_class’ and ‘output_range’ (for any acceptable
    output or a certain clas and inside a certain range). e.g. for testing
    Kernel#rand:

    [{:input => [1], :output => 0},
    {:input => [3], :output_in => [0, 1, 2]},
    {:input => , :output_class => Float,
    :output_range => 0…1.0}]

What about :in :out :fail :error

Basically, I’m seeing assert_follows_spec (or assert_calls, or whatever)
as simply an aggregate of lots of simpler asserts (assert_equals,
assert_raises, etc) as one big assert. The difference is, it’s shorter
(it saves some keystrokes), nicer/tidier, and the “spec” itself is a
data structure and can be changed/manipulated/displayed/etc more easily.

Of course, this “spec” in many cases cannot cover all aspects of the
testing, only simpler ones. So perhaps a better name for “spec” is desired.

What about ‘assert_array’ as name… it sort of indicates that there
are multiple element to compare against.

def func(val)
val + val
end
expected = [
{ :in=>1, :out=>2 },
{ :in=>‘a’, :out=>‘aa’ },
]
assert_array(expected, :func)

Or perhaps

assert_inout(expected, :func)

···

–
Simon Strandgaard

8

Eric Hodel wrote:

The assert_follows_spec() could even perform several iterations of tests
if wanted, to test deterministic-ness.

If you’ve got a spec like this, why not write a method to autogenerate
test methods?

Why is that better than something like assert_follows_spec()? It’s not
more readable IMO, it’s not shorter, it adds an additional step in the
Makefile/build process.

···

–
dave

9

This is definitely nice to me. Additionaly I don’t like so much the idea of large distance between the code and the test code. But in your scheme a least part of the test code is close, right before the tested code, that’s good I think.

“xxx # => something” is apparently a well accepted convention to describe the expected result of some code. I guess that some “standard” is needed to fully describe the expected result beyond simple cases.

I suspect that the error msg could be improved slightly, you most probably want to know the exact file/lineno where the assert failed. This may require some changes in test/unit that you probably will propose once your code is complete.

Keep me posted.

Yours,

Jean-Hugues Robert

···

At 20:38 29/05/2004 +0900, Florian wrote:

JeanHuguesRobert: About the generation of HTML documentation, I feel, again, that the
full spec is not easy to grab. I would rather like to see some RDoc
tool that would consider test cases as “examples”.

I’m currently embedding test cases into the documentation. They look a
lot like example code. I think this is commonly called “Specification by
example”. I’ve written a tool for automatically extracting and running
them. Here is an example:

Matches the space between words.

This is a zero-width anchor.

re = Regexp::English.word_boundary

text = “Hello World!”

text.gsub(re, “*”) # => “Hello World!”

def word_boundary(…
If this test case where to fail you would get an useful error like this:

  1. Failure:
    test_word_boundary()
    [(eval):3:in eval' (eval):7:in test_word_boundary’
    (eval):3:in test_word_boundary']: c:/ruby/lib/ruby/1.8/test/unit/assertions.rb:32:in assert_block’<“"Hello World!"”> expected but was <“"Hello World!"”>.

Regards,
Florian Gross

Web: http://hdl.handle.net/1030.37/1.1
Phone: +33 (0) 4 92 27 74 17

10

Simon Strandgaard wrote:

[{:input => [1], :output => 0},
{:input => [3], :output_in => [0, 1, 2]},
{:input => , :output_class => Float,
:output_range => 0…1.0}]

What about :in :out :fail :error

:in and :out are fine with me. As for the other perhaps we should mirror
more closely with the other assert_*() method names, e.g. :raises
(assert_raises), etc.

assert_array(expected, :func)
assert_inout(expected, :func)

Both are better than assert_follows_spec I think :slight_smile:

···

–
dave

11

just a question: what if you need many tests per method/class?

···

il Sat, 29 May 2004 13:34:45 +0200, Florian Gross flgr@ccan.de ha scritto::

I’m currently embedding test cases into the documentation. They look a
lot like example code. I think this is commonly called “Specification by
example”. I’ve written a tool for automatically extracting and running
them. Here is an example:

12

No additional steps. Ruby does it for you:

require 'test/unit'

def my_meth(args)
  args.inject 0 do |acc, item|
    acc + item
  end
end

class MyTest < Test::Unit::TestCase

  SPEC_ONE = [[1, 1, 1], 3]
  SPEC_TWO = [[8, 1, 2], 11]
  SPEC_THREE = [[8, 3, 4], 15]
  SPEC_FOUR = [[4, 3, 2], 9]

  constants.each do |name|
    next unless name =~ /^SPEC_(.*)/

    eval "def test_#{$1.downcase}
      input = self.class.const_get(\"#{name}\")[0]
      output = self.class.const_get(\"#{name}\")[1]
      assert_equal output, my_meth(input)
    end"
  end
end

···

David Garamond (lists@zara.6.isreserved.com) wrote:

Eric Hodel wrote:
>>The assert_follows_spec() could even perform several iterations of tests
>>if wanted, to test deterministic-ness.
>
>If you've got a spec like this, why not write a method to autogenerate
>test methods?

Why is that better than something like assert_follows_spec()? It's not
more readable IMO, it's not shorter, it adds an additional step in the
Makefile/build process.

--
Eric Hodel - drbrain@segment7.net - http://segment7.net
All messages signed with fingerprint:
FEC2 57F1 D465 EB15 5D6E 7C11 332A 551C 796C 9F04

13

Jean-Hugues ROBERT wrote:

This is definitely nice to me. Additionaly I don’t like so much the idea of large distance between the code and the test code. But in your scheme a least part of the test code is close, right before the tested code, that’s good I think.

My current gripe with unit test code is that they’re the ugliest code I
write :-). There are lots of duplication (from copy-and-paste),
no/little modularization, full of quick ‘n’ dirty hacks, and I write
them often without thinking about maintenance at all.

I think this is also true for most unit tests out there. If you want to
look at the worst spaghetti code, look at the unit test :slight_smile:

···

–
dave

14

David Garamond wrote:

assert_array(expected, :func)
assert_inout(expected, :func)

Both are better than assert_follows_spec I think :slight_smile:

How about: assert_examples

···

–
– Jim Weirich jim@weirichhouse.org http://onestepback.org

“Beware of bugs in the above code; I have only proved it correct,
not tried it.” – Donald Knuth (in a memo to Peter van Emde Boas)

15

gabriele renzi wrote:

just a question: what if you need many tests per method/class?

Many tests or many assertions?

16

Eric Hodel wrote:

If you've got a spec like this, why not write a method to autogenerate
test methods?

it adds an additional step in the Makefile/build process.

No additional steps. Ruby does it for you:

require 'test/unit'

def my_meth(args)
  args.inject 0 do |acc, item|
    acc + item
  end
end

class MyTest < Test::Unit::TestCase

  SPEC_ONE = [[1, 1, 1], 3]
  SPEC_TWO = [[8, 1, 2], 11]
  SPEC_THREE = [[8, 3, 4], 15]
  SPEC_FOUR = [[4, 3, 2], 9]

  constants.each do |name|
    next unless name =~ /^SPEC_(.*)/

    eval "def test_#{$1.downcase}
      input = self.class.const_get(\"#{name}\")[0]
      output = self.class.const_get(\"#{name}\")[1]
      assert_equal output, my_meth(input)
    end"
  end
end

Note that you don't need eval() to do this:

def my_meth(*args)
   args.inject(0) do |state, item|
     state + item
   end
end

class MyTest < Test::Unit::TestCase
   # This is an Array of Arrays of Arrays -- we should really use
   # something with more implicit semantics instead.
   Specs = [
     [[8, 1, 1], 3],
     [[8, 1, 2], 11],
     [[8, 3, 4], 15],
     [[4, 3, 2], 9]
   ]

   Specs.each_with_index do |(input, output), index|
     define_method(:"test_spec_#{index}") do
       assert_equal(output, my_meth(*input))
     end
   end
end

Regards,
Florian Gross

17

assert_cases?

assert_test_cases?

“Jim Weirich” jim@weirichhouse.org wrote in message
news:40B9199D.8010108@weirichhouse.org…

···

David Garamond wrote:

assert_array(expected, :func)
assert_inout(expected, :func)

Both are better than assert_follows_spec I think :slight_smile:

How about: assert_examples

18

Mh… I’d say tests.
I believe that, once you find a bug in , say, a method, you should add
a test for it after you fix it. Probably the test is just one
assertion, anyway. What happens to the doctests ? do you let them grow
endlessy?

···

il Sun, 30 May 2004 11:48:03 +0200, Florian Gross flgr@ccan.de ha scritto::

gabriele renzi wrote:

just a question: what if you need many tests per method/class?

Many tests or many assertions?

19

assert_matrix?

Guillaume.

···

Le 29 mai 04, à 21:33, Its Me a écrit :

assert_cases?

assert_test_cases?

“Jim Weirich” jim@weirichhouse.org wrote in message
news:40B9199D.8010108@weirichhouse.org…

David Garamond wrote:

assert_array(expected, :func)
assert_inout(expected, :func)

Both are better than assert_follows_spec I think :slight_smile:

How about: assert_examples

20

gabriele renzi wrote:

just a question: what if you need many tests per method/class?
Many tests or many assertions?
Mh… I’d say tests.
I believe that, once you find a bug in , say, a method, you should add
a test for it after you fix it. Probably the test is just one
assertion, anyway. What happens to the doctests ? do you let them grow
endlessy?

No, I think you should only put tests in there that also have a strong
purpose as examples. I think it’s actually okay to have those tests that
are cloaked as example code in the documentation and stuff that does
deeper testing in regular test suits.

But it might also make sense to put all of them into the documentation
so that the user knows for sure what belongs to the interface of the
method (the things that are tested for) and what’s implementation.
(Unspecified behavior, the things that aren’t tested for.)