[ANN] celsoft.com/Battery 0.1.1

Battery is a unit testing framework for Ruby. It captures all standard error
and output and reports the entire summary of all tests formatted as valid
YAML, for easier reading and parsing. Another key feature is that all tests
run in the order they are added to their batteries, rather than arbitrarily.

See the celsoft.com/Battery homepage for more information and documentation.

Homepage: http://battery.rubyforge.org/
Download: http://rubyforge.org/frs/?group_id=268&release_id=531

  Sean O'Dell

Hi --

···

On Sat, 12 Jun 2004, Sean O'Dell wrote:

Battery is a unit testing framework for Ruby. It captures all standard error
and output and reports the entire summary of all tests formatted as valid
YAML, for easier reading and parsing. Another key feature is that all tests
run in the order they are added to their batteries, rather than arbitrarily.

You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.

David

--
David A. Black
dblack@wobblini.net

<nasty emails snipped>

/me grabs everyone by the ear...

First of all, everyone settle down. This list has been extremely
friendly in the past, I don't want to see it degenerate into a
flamefest. At least, not unless we're flaming Perl or Python.
KIDDING!

Anyway, I don't really care about the definition of the word
"arbitrary" and whether or not it was used correctly. The tests in
TestUnit run in alphabetical order and to me that's just the way it
is.

I ran into this a while back and my initial solution was to append 01,
02, etc, to the method name to get it to run in the order I wanted,
e.g. test_01_foo, test_02_bar.

As I've gotten into unit testing I've realized something. It changes
the way you think about writing code, and thus the way you *design*
code. When I write methods, one of the things I consider is, "how
easy will it be to test this method by itself". This leads into why
people think that the order your unit tests will run shouldn't matter
- each method should be testable on its own *to make it easier to
test*, which in turn leads to smaller chunks of code, which in turn
leads to easier maintainability (in theory). It's not "just because".
I don't know if I'm making sense here, so let me quote a paragraph
from "Learning to Love Unit Testing" by Dave Thomas and Andy Hunt,
STQE, Vol 4, Issue 1, p.38:

"..., tests affect the *design* of your code. When developers get
into the habit of unit testing everything, they discover that some of
what they write is hard to test. Sometimes they find themselves
having to construct elaborate frameworks to test a single method.
Sometimes they find that the method is hard to test because it handles
many different cases. And sometimes things are hard to test because
the code that does the real work is buried deep in the program's
guts. Are these reasons not to test? On the contrary, this kind of
discovery is one of the most valuable effects of testing, because the
tests are telling you secrets about the structure of the code. List
to the tests and refactor (a fancy way of saying restructure)
accordingly. The code will be better for it. Methods will have
clearly defined functions. The class hierarchies are flatter, and
they more accurately reflect the business value of the application.
there's less coupling between classes. All of this makes the code
easier to write and simpler to maintain."

Back to ordering the unit tests. Sometimes this is probably
unavoidable. If that's the case, so be it. The unit testing police
aren't going to come get you. Just make sure you document it
somewhere. However, when you come to a point where you think you'll
need your tests in a certain order, you should at least consider
whether a Mock Object wouldn't be a better solution.

What does this have to do with Battery? Man, I don't know. I lost
track of why I started on this thread about 2 paragraphs ago. :wink:

Dan

David A. Black writes:

Hi --

> Battery is a unit testing framework for Ruby. It captures all standard error
> and output and reports the entire summary of all tests formatted as valid
> YAML, for easier reading and parsing. Another key feature is that all tests
> run in the order they are added to their batteries, rather than arbitrarily.

You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.

Is that guaranteed by the API or should I not rely on it?

Jim

···

On Sat, 12 Jun 2004, Sean O'Dell wrote:

--
Jim Menard, jimm@io.com, http://www.io.com/~jimm/
"COGITO, EGGO SUM." I think, therefore I am a waffle.
   -- .sig of Mr. Ska on Slashdot.org

When tests are run alphabetically, it makes it tough to run the same tests in
specific orders. I wanted to be able to take a single test, and run it
combined with other tests in varying orders. celsoft.com/Battery lets me
create batteries in which I add the same test multiple times but in the order
I choose.

  Sean O'Dell

···

On Friday 11 June 2004 11:47, David A. Black wrote:

Hi --

On Sat, 12 Jun 2004, Sean O'Dell wrote:
> Battery is a unit testing framework for Ruby. It captures all standard
> error and output and reports the entire summary of all tests formatted as
> valid YAML, for easier reading and parsing. Another key feature is that
> all tests run in the order they are added to their batteries, rather than
> arbitrarily.

You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.

It is guaranteed. Even if other orderings are added at some point in the future, running in alphabetical order will always be supported.

Nathaniel
Terralien, Inc.

<:((><

···

On Jun 11, 2004, at 13:50, Jim Menard wrote:

David A. Black writes:

Hi --

On Sat, 12 Jun 2004, Sean O'Dell wrote:

Battery is a unit testing framework for Ruby. It captures all standard error
and output and reports the entire summary of all tests formatted as valid
YAML, for easier reading and parsing. Another key feature is that all tests
run in the order they are added to their batteries, rather than arbitrarily.

You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.

Is that guaranteed by the API or should I not rely on it?

> You're implying, I believe, that Test::Unit runs the test methods in
> arbitrary order. It doesn't; it runs them in alphabetical order.

Is that guaranteed by the API or should I not rely on it?

Beside your point a bit here, but you shouldn't rely on tests running
in ANY order.

Hi --

···

On Sat, 12 Jun 2004, Sean O'Dell wrote:

On Friday 11 June 2004 11:47, David A. Black wrote:
> Hi --
>
> On Sat, 12 Jun 2004, Sean O'Dell wrote:
> > Battery is a unit testing framework for Ruby. It captures all standard
> > error and output and reports the entire summary of all tests formatted as
> > valid YAML, for easier reading and parsing. Another key feature is that
> > all tests run in the order they are added to their batteries, rather than
> > arbitrarily.
>
> You're implying, I believe, that Test::Unit runs the test methods in
> arbitrary order. It doesn't; it runs them in alphabetical order.

When tests are run alphabetically, it makes it tough to run the same tests in
specific orders. I wanted to be able to take a single test, and run it
combined with other tests in varying orders. celsoft.com/Battery lets me
create batteries in which I add the same test multiple times but in the order
I choose.

I wasn't talking about the merits of one design or the other, just
wanted to clarify the point that in Nathaniel's design the tests are
not run arbitrarily.

David

--
David A. Black
dblack@wobblini.net

Nathaniel Talbott said:

···

On Jun 11, 2004, at 13:50, Jim Menard wrote:

David A. Black writes:

You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.

Is that guaranteed by the API or should I not rely on it?

It is guaranteed. Even if other orderings are added at some point in
the future, running in alphabetical order will always be supported.

Even if it is guaranteed, I recommend not relying on it in the tests
themselves. Each test should be very loosely coupled to its siblings and
the order should not matter.

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

Arbitrary meaning that, from a library USER'S viewpoint, the decision to order
tests alphabetically is without reason and seems hastily settled upon.
Perhaps the library developer had a good reason, but as a user of the library
with different needs, I don't see/concur with the reason.

Anyway, if you like it alphabetical, you got one library. If you like to
order tests yourself, now you have another.

  Sean O'Dell

···

On Friday 11 June 2004 18:08, David A. Black wrote:

On Sat, 12 Jun 2004, Sean O'Dell wrote:
> On Friday 11 June 2004 11:47, David A. Black wrote:
> > Hi --
> >
> > On Sat, 12 Jun 2004, Sean O'Dell wrote:
> > > Battery is a unit testing framework for Ruby. It captures all
> > > standard error and output and reports the entire summary of all tests
> > > formatted as valid YAML, for easier reading and parsing. Another key
> > > feature is that all tests run in the order they are added to their
> > > batteries, rather than arbitrarily.
> >
> > You're implying, I believe, that Test::Unit runs the test methods in
> > arbitrary order. It doesn't; it runs them in alphabetical order.
>
> When tests are run alphabetically, it makes it tough to run the same
> tests in specific orders. I wanted to be able to take a single test, and
> run it combined with other tests in varying orders. celsoft.com/Battery
> lets me create batteries in which I add the same test multiple times but
> in the order I choose.

I wasn't talking about the merits of one design or the other, just
wanted to clarify the point that in Nathaniel's design the tests are
not run arbitrarily.

The order shouldn't matter, but it's better to have control over the order
rather than have them run in an arbitrary order selected by the library.
With control, you could randomize the list of tests and run the randomized
battery n times. When the tests are alphabetical, they're always run in
alphabetical order.

  Sean O'Dell

···

On Friday 11 June 2004 13:01, Jim Weirich wrote:

Nathaniel Talbott said:
> On Jun 11, 2004, at 13:50, Jim Menard wrote:
>> David A. Black writes:
>>> You're implying, I believe, that Test::Unit runs the test methods in
>>> arbitrary order. It doesn't; it runs them in alphabetical order.
>>
>> Is that guaranteed by the API or should I not rely on it?
>
> It is guaranteed. Even if other orderings are added at some point in
> the future, running in alphabetical order will always be supported.

Even if it is guaranteed, I recommend not relying on it in the tests
themselves. Each test should be very loosely coupled to its siblings and
the order should not matter.

Yes but I can think of a very simple reason someone might want to reorder there
tests a bit. Right now there is a bug in ruby 1.8.2 that causes one of the unit
tests to segfault the interpreter when running an extension test. As a result
the test suite never completes, so I only know what works before it. If this
happened in a unit test that was of my own product there might exist a case in
which I wanted to double check all my others tests before the complete failure
case. Particularly if I was writing more specific unit tests to try to figure
out what was causing the segfault.

I might also just want the reports on related components to be in the same area
when they fail if I change something.

Clearly you shouldn't set up tests to depend on ordering but for your own
understanding of the output I can think of lots of reasons you might want your
own ordering.

Charlie

···

On Sat, 12 Jun 2004, Jim Weirich wrote:

Nathaniel Talbott said:
> On Jun 11, 2004, at 13:50, Jim Menard wrote:
>> David A. Black writes:
>>> You're implying, I believe, that Test::Unit runs the test methods in
>>> arbitrary order. It doesn't; it runs them in alphabetical order.
>>
>> Is that guaranteed by the API or should I not rely on it?
>
> It is guaranteed. Even if other orderings are added at some point in
> the future, running in alphabetical order will always be supported.

Even if it is guaranteed, I recommend not relying on it in the tests
themselves. Each test should be very loosely coupled to its siblings and
the order should not matter.

Have you checked the archives, or asked Nathaniel? That might help
you see the reasoning behind the work (including input from the
community, Nathaniel's attention to which is utterly exemplary), and
then decide in a more informed manner whether or not you concur with
it.

Also, now that you've been made aware of the fact that Test::Unit runs
the tests in alphabetical order, it would be much more collegial not
to cling to the use of the word "arbitrary" and its derivatives in an
effort to make retroactive sense of your original, incorrect claim.
I've watched Nathaniel work on Test::Unit over the years, and heard
him talk about it online and in person, and I can assure you that
there is nothing arbitrary about his design whatsoever. Anyone
writing a testing framework in Ruby, whether they make the same
decisions he has or not, should be grateful or at least gracious.

David

···

On Sat, 12 Jun 2004, Sean O'Dell wrote:

Arbitrary meaning that, from a library USER'S viewpoint, the decision to order
tests alphabetically is without reason and seems hastily settled upon.
Perhaps the library developer had a good reason, but as a user of the library
with different needs, I don't see/concur with the reason.

--
David A. Black
dblack@wobblini.net

How is alphabetical order any better/worse than "the order they are
added to their batteries" in this regard? Or does your framework
allow order randomization?

···

On Sat, 12 Jun 2004 05:09:09 +0900, Sean O'Dell <sean@celsoft.com> wrote:

The order shouldn't matter, but it's better to have control over the order
rather than have them run in an arbitrary order selected by the library.
With control, you could randomize the list of tests and run the randomized
battery n times. When the tests are alphabetical, they're always run in
alphabetical order.

Nathaniel Talbott said:

David A. Black writes:

You're implying, I believe, that Test::Unit runs the test methods in
arbitrary order. It doesn't; it runs them in alphabetical order.

Is that guaranteed by the API or should I not rely on it?

It is guaranteed. Even if other orderings are added at some point in
the future, running in alphabetical order will always be supported.

Even if it is guaranteed, I recommend not relying on it in the tests
themselves. Each test should be very loosely coupled to its siblings and
the order should not matter.

Yes but I can think of a very simple reason someone might want to reorder there
tests a bit. Right now there is a bug in ruby 1.8.2 that causes one of the unit
tests to segfault the interpreter when running an extension test. As a result
the test suite never completes, so I only know what works before it. If this
happened in a unit test that was of my own product there might exist a case in
which I wanted to double check all my others tests before the complete failure
case. Particularly if I was writing more specific unit tests to try to figure
out what was causing the segfault.

In your own tests you can always put
flunk("To be fixed later")

or something like that in the beginning of the segfaulting test or rename the test so that it does not run at all (I usually put underscore in front of the test name). This does not justify introducing the test order at all. There might be other reasons, though. I simply haven't heard of any good one yet.

By the way, with Test::Unit you have ability to specify the exact test method or a test class on the command line, even using regexps. There's a bug in test specification though, however it is easily fixed. I did not submit usual report, sorry. With this feature, if I have a huge test suite, I can concentrate on a particular problem I am working on right now, and when it finally passes run the whole suite, happily leaning at the back of my chair.

I might also just want the reports on related components to be in the same area
when they fail if I change something.

Clearly you shouldn't set up tests to depend on ordering but for your own
understanding of the output I can think of lots of reasons you might want your
own ordering.

Charlie

Sincerely,
Gennady Bystritsky

···

On Jun 11, 2004, at 9:48 PM, Charles Comstock wrote:

On Sat, 12 Jun 2004, Jim Weirich wrote:

On Jun 11, 2004, at 13:50, Jim Menard wrote:

ar·bi·trar·y
adj.
Determined by chance, whim, or impulse, and not by necessity, reason, or
principle.

There is no fundamental reason tests should be ordered alphabetically. I've
demonstrated that it's only too easy to order tests any way I want, so I know
alphabetic is not out of necessity. So then, what reason or principle
determines that tests should be run alphabetically? I see none. Is there a
reason and/or principal that says they must be run with that order? Please
tell me if there is, and I'll retract.

  Sean O'Dell

···

On Saturday 12 June 2004 03:49, David A. Black wrote:

On Sat, 12 Jun 2004, Sean O'Dell wrote:
> Arbitrary meaning that, from a library USER'S viewpoint, the decision to
> order tests alphabetically is without reason and seems hastily settled
> upon. Perhaps the library developer had a good reason, but as a user of
> the library with different needs, I don't see/concur with the reason.

Have you checked the archives, or asked Nathaniel? That might help
you see the reasoning behind the work (including input from the
community, Nathaniel's attention to which is utterly exemplary), and
then decide in a more informed manner whether or not you concur with
it.

Also, now that you've been made aware of the fact that Test::Unit runs
the tests in alphabetical order, it would be much more collegial not
to cling to the use of the word "arbitrary" and its derivatives in an
effort to make retroactive sense of your original, incorrect claim.
I've watched Nathaniel work on Test::Unit over the years, and heard
him talk about it online and in person, and I can assure you that
there is nothing arbitrary about his design whatsoever. Anyone
writing a testing framework in Ruby, whether they make the same
decisions he has or not, should be grateful or at least gracious.

Tests are stored as an array of blocks and are called in the order they appear
in the array. I just now added an accessor method to the @tests instance
variable so you can arrange the order anyway you want..

But even before that change, you could create a number of batteries and call
battery#add_test in any order you prefer to create your own run order for
each battery, then call battery#run for each, using a common report object,
and get one big final report for it all.

  Sean O'Dell

···

On Friday 11 June 2004 13:19, Michael Campbell wrote:

On Sat, 12 Jun 2004 05:09:09 +0900, Sean O'Dell <sean@celsoft.com> wrote:
> The order shouldn't matter, but it's better to have control over the
> order rather than have them run in an arbitrary order selected by the
> library. With control, you could randomize the list of tests and run the
> randomized battery n times. When the tests are alphabetical, they're
> always run in alphabetical order.

How is alphabetical order any better/worse than "the order they are
added to their batteries" in this regard? Or does your framework
allow order randomization?

arbitrary:
Determined by chance, whim, or impulse, and not by necessity, reason, or
principle.

There is no fundamental reason tests should be ordered alphabetically.

Maybe, maybe not -- but in either case, I was addressing your original
claim, which was not that choosing alphabetical order over other designs
was arbitrary, but rather that in Nathaniel's design, the tests were
run in arbitrary (i.e., determined by chance, whim, etc.) order. They
aren't, so I wanted to let you and other readers know.

This had (and has) nothing to do with the processing of choosing
alphabetical order (which you introduced into the discussion later),
but rather with the fact that alphabetical order is not, itself,
arbitrary -- and, concommitantly, with the fact that Nathaniel didn't
deserve to have his design misrepresented as involving something it
doesn't.

David

···

--

David A. Black
dblack@wobblini.net

Early in test/unit's life, test order was arbitrary (well, OK, they were actually run in the order that Module#public_instance_methods returned them... but that was pretty arbitrary). In general, this was not a bad thing, because unit tests that are dependent on test order are a pretty serious code smell. However, PragDave pointed out that adding a bit more predictability would be a good thing, so I started sorting the methods alphabetically. This allows one to order test methods via naming 'hacks', but they feel like hacks. Some people don't like that, but I consider it to be a feature, because THEY ARE hacks.

So yes, I can agree that sorting alphabetically is a fairly arbitrary decision (although I can't think of another order I would prefer in its place). However, the choice to not make ordering an important part of test/unit's function is not arbitrary. The only other ordering I have really considered adding random order, with the seed being printed out each run so that results can be duplicated. That would be useful for removing inter-test dependencies, while most other orderings I can think of would encourage them.

As I've watched the uses of test/unit grow and change over time, I've also considered adding more ability to run tests in specific orders... however, this is because I see test/unit being used in various places at an acceptance (or customer) testing level, where those needs are much more legitimate. It's a challenge to introduce something like that, though, and still encourage good unit testing practices. Perhaps it's time for test/accept?

Anyhow, I'm glad that Sean has released this new library... it will only make test/unit better (I've already gleaned a few ideas from it). And when I get a chance to overhaul the test/unit documentation, part of the overhaul will be an attempt to communicate not just what test/unit does, but also what it doesn't do, and why.

Nathaniel
Terralien, Inc.

<:((><

···

On Jun 12, 2004, at 12:28, Sean O'Dell wrote:

ar·bi·trar·y
adj.
Determined by chance, whim, or impulse, and not by necessity, reason, or
principle.

There is no fundamental reason tests should be ordered alphabetically. I've
demonstrated that it's only too easy to order tests any way I want, so I know
alphabetic is not out of necessity. So then, what reason or principle
determines that tests should be run alphabetically? I see none. Is there a
reason and/or principal that says they must be run with that order? Please
tell me if there is, and I'll retract.

When you're given no choice, as far as you're concerned as a library user,
they're run in an arbitrary order. The fact that it turns out they are run
alphabetically doesn't change the fact that the system of ordering is
arbitrary. There is no rhyme, reason or principal determining the order; it
was decided by the author at his prerogative. If the tests were run
according to the length of the method names, that would still be an arbitrary
order. As far as I'm concerned, the order that tests are run is a black box.

  Sean O'Dell

···

On Saturday 12 June 2004 13:34, David A. Black wrote:

> arbitrary:
> Determined by chance, whim, or impulse, and not by necessity, reason, or
> principle.
>
> There is no fundamental reason tests should be ordered alphabetically.

Maybe, maybe not -- but in either case, I was addressing your original
claim, which was not that choosing alphabetical order over other designs
was arbitrary, but rather that in Nathaniel's design, the tests were
run in arbitrary (i.e., determined by chance, whim, etc.) order. They
aren't, so I wanted to let you and other readers know.

This had (and has) nothing to do with the processing of choosing
alphabetical order (which you introduced into the discussion later),
but rather with the fact that alphabetical order is not, itself,
arbitrary -- and, concommitantly, with the fact that Nathaniel didn't
deserve to have his design misrepresented as involving something it
doesn't.