Test::Unit -> order of tests?

ruby-talk
21

“Nathaniel Talbott” nathaniel@talbott.ws writes:

The problem is that there is no good way to run the tests in the
order defined.

class TestCase
class << self
def method_added(symbol)
# how would matz do this Perlish test?
if symbol.id2name =~ /^test_/

Matz called “perlish” this
if /^test_/

There’s nothing “perlish” in your test. As far as I understand, Matz is
only against operations on implied operands. Sorry for an off-topic
remark.

···

On Monday, Jan 27, 2003, at 20:10 US/Pacific, Matt Armstrong wrote:

@test_methods ||=
@test_methods << symbol
end
end
attr_reader :test_methods
end
end

class MyTestCase < TestCase
def test_zzz; end
def test_mmm; end
def test_aaa; end
end

class MyTestCase2 < TestCase
def test_zzz2; end
def test_mmm2; end
def test_aaa2; end
end

p MyTestCase.test_methods
#=> [:test_zzz, :test_mmm, :test_aaa]
p MyTestCase2.test_methods
#=> [:test_zzz2, :test_mmm2, :test_aaa2]

Anyhow, on running tests in (pseudo-)random order, that’s one of the
test orderings I’d like to implement in the future. The only problem
with it is, if things fail when you’re running them randomly, you’ll
want to know what order they were run in that particular time so you
can fix the problem. Otherwise it might be more frustrating than
helpful.

I can think of 2 solutions here:

- Test::Unit supports saving the order of the tests it runs in a
  text file, and can parse that file and run the tests in the same
  order.
- Test::Unit includes its own random number generator, spits
  out the random seed it is using, and can support seeding its RNG
  to a particular number.  (you can't use Ruby's RNG since tests
  themselves might use it)
22

It occurs to me that you might to support unordered tests by providing a
“random” test sequence method.

···

On Tue, Jan 28, 2003 at 10:56:02AM +0900, Nathaniel Talbott wrote:

I hate the fact that test ordering is ever important. I hope that people
only ever use different test orderings in order to identify test
interdependency as a step to removing it. But I recognize that there are
various reasons (a few of them even pragmatic :wink: that will cause
ordering to matter for some tests.

…snip

–
Alan Chen
Digikata Computing
http://digikata.com

23

Anyhow, on running tests in (pseudo-)random order, that’s one of the

test orderings I’d like to implement in the future. The only problem

with it is, if things fail when you’re running them randomly, you’ll

want to know what order they were run in that particular time so you

can fix the problem. Otherwise it might be more frustrating than
helpful.

Emit the random number seed so that it is repeatable,

Excellent idea. As Matt says, I’d have to insulate myself from the
standard random number generator, but that doesn’t seem too hard to do.

and
give a verbose option (may be there already?) that emits the
method names as they are tested. That oughta do it.

The Console::TestRunner in CVS supports several new output levels,
including one that outputs every test run, so we’re covered there.

Nathaniel

<:((><

···

Gavin Sinclair [mailto:gsinclair@soyabean.com.au] wrote:

RoleModel Software, Inc.
EQUIP VI

24

The problem is that there is no good way to run the tests in the
order
defined.

class TestCase
class << self
def method_added(symbol)
# how would matz do this Perlish test?
if symbol.id2name =~ /^test_/
@test_methods ||=
@test_methods << symbol
end
end
attr_reader :test_methods
end
end

class MyTestCase < TestCase
def test_zzz; end
def test_mmm; end
def test_aaa; end
end

class MyTestCase2 < TestCase
def test_zzz2; end
def test_mmm2; end
def test_aaa2; end
end

p MyTestCase.test_methods
#=> [:test_zzz, :test_mmm, :test_aaa]
p MyTestCase2.test_methods
#=> [:test_zzz2, :test_mmm2, :test_aaa2]

That’s pretty slick! I had a feeling as soon as I said it wouldn’t work
that someone would propose a solution. There might be an issue with
doing this, though - after a little IRB’ing here, it appears that
including a module will not trigger #method_added for each method of the
module. But perhaps there’s a way around that, too :slight_smile:

Also, and it could just be me being over-cautious, but adding test
methods by trapping their definition seems to be more vulnerable to
problems than waiting until it’s time to run them and then gathering
them from where I expect them to be. I don’t have to worry about missing
anything, as Ruby is collecting it all for me. But I’m a worrywart.

Anyhow, on running tests in (pseudo-)random order, that’s one of the

test orderings I’d like to implement in the future. The only problem

with it is, if things fail when you’re running them randomly, you’ll

want to know what order they were run in that particular time so you

can fix the problem. Otherwise it might be more frustrating than
helpful.

I can think of 2 solutions here:

- Test::Unit supports saving the order of the tests it runs in a
  text file, and can parse that file and run the tests in the same
  order.
- Test::Unit includes its own random number generator, spits
  out the random seed it is using, and can support seeding its RNG
  to a particular number.  (you can't use Ruby's RNG since tests
  themselves might use it)

I think I like option #2 best, as it appears at first blush to be
simplest to provide.

Finally, as there seems to be so much interest in this, I’ll mentally
bump it up a few levels in priority.

Thanks to everyone for their excellent input,

Nathaniel

<:((><

···

Matt Armstrong [mailto:matt@lickey.com] wrote:

RoleModel Software, Inc.
EQUIP VI

25

nathaniel@NOSPAMtalbott.ws writes:

The problem is that there is no good way to run the tests in the
order
defined.

class TestCase
class << self
def method_added(symbol)

[…]

end
attr_reader :test_methods

end
end

[…]

That’s pretty slick! I had a feeling as soon as I said it wouldn’t
work that someone would propose a solution. There might be an issue
with doing this, though - after a little IRB’ing here, it appears
that including a module will not trigger #method_added for each
method of the module. But perhaps there’s a way around that, too :slight_smile:

Ooh, yeah, yuck. The included module won’t have a list of methods in
declaration order anyway. You’d have to a method_added hook on Object
or Module and track all classes just in case they are included in a
TestCase – not pretty.

Also, and it could just be me being over-cautious, but adding test
methods by trapping their definition seems to be more vulnerable to
problems than waiting until it’s time to run them and then
gathering them from where I expect them to be. I don’t have to worry
about missing anything, as Ruby is collecting it all for me. But I’m
a worrywart.

Yes, it requires Test::Unit to become more involved with Ruby’s
internal guts (method_added, etc.) than makes me comfortable too.

Alphabetical order is simple to define, easily verifiable, and fits in
most people’s heads.

···

Matt Armstrong [mailto:matt@lickey.com] wrote:

26

Nevermind – Note to self, must remember to read all branches of the thread before posting…

···

On Tue, Jan 28, 2003 at 02:38:58PM +0900, Alan Chen wrote:

It occurs to me that you might to support unordered tests by providing a
“random” test sequence method.

–
Alan Chen
Digikata Computing
http://digikata.com

27

Something I’ve wished for: I’d like my tests to be ordered such that
the tests that failed last time I ran the tests are run first.

-Tom

···

On Tue, Jan 28, 2003 at 02:38:58PM +0900, Alan Chen wrote:

It occurs to me that you might to support unordered tests by providing a
“random” test sequence method.

On Tue, Jan 28, 2003 at 10:56:02AM +0900, Nathaniel Talbott wrote:

I hate the fact that test ordering is ever important. I hope that people
only ever use different test orderings in order to identify test
interdependency as a step to removing it. But I recognize that there are
various reasons (a few of them even pragmatic :wink: that will cause
ordering to matter for some tests.

…snip

–
Alan Chen
Digikata Computing
http://digikata.com

28

There might be an issue with doing this, though - after
a little IRB’ing here, it appears that including a module
will not trigger #method_added for each method of the
module. But perhaps there’s a way around that, too :slight_smile:

Ooh, yeah, yuck. The included module won’t have a list of
methods in declaration order anyway. You’d have to a
method_added hook on Object or Module and track all classes
just in case they are included in a TestCase – not pretty.

I'll mention right off the bat that I'm a Ruby newbie, but based on an

example I saw in the Pickaxe book, it seems like you could simply put a hook
in Module that would attach an (autoincremented) ‘testcasenumber’ attribute
to any method whose name =~ /^test_/ when that method is defined. Then
collect all of the test cases when you are ready to run them (like you
currently do), but sort on the ‘testcasenumber’ attribute before running
them. Any methods not involved in testing would (theoretically) not be
hindered by the additional attribute.

- Warren Brown