Minitest randomization

ruby-talk
1

Minitest was intended to be a minimal replacement for Test::Unit and RSpec. It mostly succeeds in this goal; however, it violates it in one annoying case. It adds a feature not found in RSpec, a feature that I find noble in theory, but troublesome in practice: test randomization. And it enables it by default.

This has no practical effect on an all-succeeding suite, or one in which a single test is failing. But if you have a few failing tests, it completely messes with your mind. Usually you will focus in on a single failing test and work to make that one pass; usually this is the first one to fail. But the randomization means that when you run it and watch it fail again you have to squint and read through all the randomly ordered failures to see where your current victim is. It's a real flow killer.

Randomization also mandates spewing options (notably seed) onto the console, which serve only to clutter up the display and obscure the all-important success or failure methods. And for some reason you're printing them twice (once at the beginning and again at the end of the run). And even if knowing the seed is important if it provoked a failure, why bother printing them during a successful run?

Moreover, most well-factored OO code these days does not exhibit isolation problems. Yes, it's a good idea to randomize your tests once in a while -- say, once before checking in, or before a release -- but the benefit is not worth the above costs. And even if I thought it were worth it, an alleged functional replacement library should not go changing default behavior like that.

If I submit a patch to do the following, will you accept it?

* make test randomization an option ("randomize")
* make the default for that option be "off"
* output the command-line options only if one of the following is true:
** "verbose" is on
** "randomize" is on AND there was a failure

Cheers -

- Alex (the Wrong guy :-))

2

The best solution would be to make the execution random, but the
output in order.

3

This has no practical effect on an all-succeeding suite, or one in which a single test is failing.

Really? I think preventing test order dependency has a very practical effect.

If I submit a patch to do the following, will you accept it?

* make test randomization an option ("randomize")

It is an option on a class by class basis. See `ri MiniTest::Unit::TestCase::test_order`.

* make the default for that option be "off"

I actually disagree with this, but I think that is less important atm.

Why aren't you using --seed when you rerun your specs? If you use --seed with the previous value, all of your complaint about having to "squint" to find your previous failure goes away.

Phrogz contributed some patches for 1.6.0 that ensured that specs could be run in defined order by serializing their names. Combine that with ::test_order above and you have exactly what you want. Easy-peasy!

I could have _sworn_ that Spec already overrode ::test_order to be :sorted, but I have absolutely no commit to that effect. I guess Phrogz was explicitly defining that on his specs.

I still think that random tests/specs are stronger tests/specs and completely disagree with you that "most well-factored OO [test] code these days does not exhibit isolation problems" on the basis that most OO [test] code is not well-factored. minitest's test dir flays at 535. Wrong's tests flay at 1150.

Anecdotally (unfortunately, nothing I can go into in great detail), I've seen far too many projects with test order dependencies, which is the reason that feature went into the library in the first place.

In the cases where I'm doing big refactorings and getting huge swaths of errors and the randomization is bugging me, I'll temporarily pop in test_order to sort them and get through my work. But I always remove it before committing.

* output the command-line options only if one of the following is true:
** "verbose" is on
** "randomize" is on AND there was a failure

Unfortunately, we need to output the seed value at the beginning in the case that your tests not only fail, but crash (like when you're Aaron Patterson and you're working on C extensions instead of writing ruby like a good person).

I can see it going the verbose route, but maybe there needs to be a middle level verbosity? Full verbosity is damn noisy/annoying unless you need those test method times (and then it is _awesome_).

···

On Oct 6, 2010, at 15:03 , Alex Chaffee wrote:

4

that's not a bad idea at all... but it would be DAMN confusing if you did have test order dependency errors but couldn't see why you were getting your failures because they always showed up in sorted order... I'll have to think about that one for a bit.

···

On Oct 6, 2010, at 16:11 , Steve Klabnik wrote:

The best solution would be to make the execution random, but the
output in order.

5

P.S. I didn't want to go there, but if you're gonna throw down --
"flay lib" for Wrong gives 168 to Minitest's 795, sucka! :slight_smile: (and they
both have about 28Kloc)

Sorry, read "wc" wrong -- they both have about 1100 lines of code.

And I just did a checkin that makes Wrong's flay 0.

Sucka. :slight_smile:

- A