Test::Unit Patch that allows test methods to be executed sequentially

ruby-talk
1

This patch to Test::Unit allows you to create TestCases whose test
methods will be executed in the order you've defined them, rather than
alphabetically, which is what Test::Unit does today.

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

Thus the foloowing TestCase will print "EFGH":

class TC2_Sequential < Test::Unit::TestCase
  execute :sequentially
  def test_b; print 'E'; end
  def test_a; print 'F'; end
  def test_d; print 'G'; end
  def test_c; print 'H'; end
end

I realize that there has been some controversy in the past about
whether is a good idea or not. This debate has hinged on issues about
proper unit testing.

However, with Watir (http://wtr.rubyforge.org), many people are now
using Test::Unit for system acceptance testing, which presents a
different context for the value of ordering tests.

My hope is that some people will find this to be useful. My thanks go
to Why's MetaClass article, which gave me the info i needed to create
this.

Bret

2

Why not subclass TestCase and create SequentialTestCase or WatirTestCase? I don't see any reason why this functionality needs to be folded into test/unit but I see your argument for Watir. We should be trying our damnedest to make test/unit thinner, not thicker.

···

On May 21, 2006, at 7:45 PM, bpettichord@gmail.com wrote:

class TC2_Sequential < Test::Unit::TestCase
  execute :sequentially
  def test_b; print 'E'; end
  def test_a; print 'F'; end
  def test_d; print 'G'; end
  def test_c; print 'H'; end
end

I realize that there has been some controversy in the past about
whether is a good idea or not. This debate has hinged on issues about
proper unit testing.

--
_why: zenspider's most intense moments of solice are immediately following the slaughter [...]
_why: that topknot's the only thing keeping a lid on the righteous anger
bricolage: yeah, that and his flagrant obsession with dvorak

3

I don't see a patch there.

···

On May 21, 2006, at 7:45 PM, bpettichord@gmail.com wrote:

This patch to Test::Unit allows you to create TestCases whose test
methods will be executed in the order you've defined them, rather than
alphabetically, which is what Test::Unit does today.

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

4

Ryan Davis said:

Why not subclass TestCase and create SequentialTestCase or
WatirTestCase? I don't see any reason why this functionality needs to
be folded into test/unit but I see your argument for Watir. We should
be trying our damnedest to make test/unit thinner, not thicker.

Unfortunately, subclassing TestCase is not a good idea. All subclasses
of TestCase are presumed to be concrete tests and you will get errors
if the subclass does not have a test method. (Try it out if you don't
believe me.) There is no provision for "Virtual" testcases, which is
what we would want here.

I think the ideal solution would be to refactor the TestCase code along
the lines of this patch, and thereby expose
TestCase.sorted_test_methods so that it could then be overlayed by
Modules.

This would keep the code from bulking up and allow users to add custom
behavior without modifying the class itself. For example, there has
been some interest expressed in randomizing the order of the test
methods. This suggestion would allow users to easily do this.

Bret

5

Ryan Davis wrote:

proper unit testing.

Why not subclass TestCase and create SequentialTestCase or
WatirTestCase?

The problem with subclassing is that it doesn't combine well with other
potential Test::Unit add-ins. (For that matter, I suspect the patch way
doesn't combine well either, but I haven't looked at the details).

For example, I have a Test::Unit::TestCase mod that allows FlexMock
objects to be automatically verified at the end of a test case. If I
make this a sub-class, then Brett can't use my FlexMock subclass with
his Watir subclass.

Rather, make the add-ins modules, then the user can mix-in whatever
behavior they need. For example ...

   class MyTest < Test::Unit::TestCase
     include FlexMock::TestCase
     include Watir::TestCase

     def test_using_both_watir_and_flexmock_extensions
       ...
     end
   end

Now, the flexmock one ties in by overriding teardown, so any user
supplied setup/teardown methods should carefully invoke super (which
they probably should do anyways).

-- Jim Weirich

···

On May 21, 2006, at 7:45 PM, bpettichord@gmail.com wrote:

--
Posted via http://www.ruby-forum.com/\.

6

Click "Download Patch." Or click the attachments tab. I also just added a
link at the bottom of the page.

I have been updating the code on this page based on comments that i get.

Here's the current version of the code:

require 'test/unit'

module Test
  module Unit
    class TestCase
      @@order = :alphabetically
      class << self
        attr_accessor :test_methods, :order
        def test_methods
          @test_methods ||=
        end
        def order
          @order || @@order
        end
        def default_order= order
          @@order = order
        end
        def sorted_test_methods
          case order
          when :alphabetically: test_methods.sort
          when :sequentially: test_methods
          when :reversed_sequentially: test_methods.reverse
          when :reversed_alphabetically: test_methods.sort.reverse
          else raise ArgumentError, "Execute option not supported:
#{@order}"
          end
        end
        def suite
          suite = TestSuite.new(name)
          sorted_test_methods.each do |test|
            catch :invalid_test do
              suite << new(test)
            end
          end
          if (suite.empty?)
            catch :invalid_test do
              suite << new(:default_test)
            end
          end
          return suite
        end
        def method_added id
          name = id.id2name
          test_methods << name if name =~ /^test./
        end
        def execute order
          @order = order
        end
      end
    end
  end
end

···

On 5/22/06, Eric Hodel <drbrain@segment7.net> wrote:

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

I don't see a patch there.

7

Ryan Davis said:

Why not subclass TestCase and create SequentialTestCase or
WatirTestCase? I don't see any reason why this functionality needs to
be folded into test/unit but I see your argument for Watir. We should
be trying our damnedest to make test/unit thinner, not thicker.

Unfortunately, subclassing TestCase is not a good idea. All subclasses
of TestCase are presumed to be concrete tests and you will get errors
if the subclass does not have a test method. (Try it out if you don't
believe me.) There is no provision for "Virtual" testcases, which is
what we would want here.

I do it all the time. The simple solution is to put a "test_dummy" in the superclass. Really not hard and a fair tradeoff to have the right code in the right place.

I think the ideal solution would be to refactor the TestCase code along
the lines of this patch, and thereby expose
TestCase.sorted_test_methods so that it could then be overlayed by
Modules.

I disagree. As I said before, we need less code, not more. (I do wholly agree that test/unit needs refactoring, but to make it thinner not thicker)

···

On May 22, 2006, at 11:31 AM, bpettichord@gmail.com wrote:

--
ryand-ruby@zenspider.com
Seattle.rb | Home - Seattle.rb
http://blog.zenspider.com/

8

I think Eric's point was that it isn't a patch.

···

On May 22, 2006, at 2:47 PM, Bret Pettichord wrote:

On 5/22/06, Eric Hodel <drbrain@segment7.net> wrote:

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

I don't see a patch there.

Click "Download Patch." Or click the attachments tab. I also just added a
link at the bottom of the page.

9

The problem with subclassing is that it doesn't combine well with other
potential Test::Unit add-ins. (For that matter, I suspect the patch way
doesn't combine well either, but I haven't looked at the details).

"potential add-ins" is a red flag for YAGNI.

For example, I have a Test::Unit::TestCase mod that allows FlexMock
objects to be automatically verified at the end of a test case. If I
make this a sub-class, then Brett can't use my FlexMock subclass with
his Watir subclass.

I guess I don't understand. Is your FlexMock mod a subclassed testcase as well? If not, then where is the conflict?

I guess where I'm getting at is this: Brett's modification is NOT a unit test extension. Your extension seems to be orthogonal to TestCase, and so belongs in a module to be mixed in wherever needed. Brett's extension isn't orthogonal. As he said, it is a modification at a system or integration level. Test::Unit is a fine execution framework but TestCase in particular has some *cough*flaws*cough* design restrictions that make subclassing a bit of a PITA and as a result, TestCase gets used for everything. This is a shame IMO as I've pointed out elsewhere (http://blog.zenspider.com/archives/2006/01/move_over_testu.html\) but we have workarounds that are satisfactory.

I want people to test. I want them to write all the unit, system, integration, and performance tests they possibly want. I want them to have the tools they need to do this, but I also want those tools to not confuse them as to what type of tests they are writing and I think his patch is confusing.

Rather, make the add-ins modules, then the user can mix-in whatever
behavior they need. For example ...

   class MyTest < Test::Unit::TestCase
     include FlexMock::TestCase
     include Watir::TestCase

     def test_using_both_watir_and_flexmock_extensions
       ...
     end
   end

Now, the flexmock one ties in by overriding teardown, so any user
supplied setup/teardown methods should carefully invoke super (which
they probably should do anyways).

*nod* I think this is a fine alternative. I'd probably take it a step further and use it AND subclassing to make things both clearer and a bit easier.

···

On May 22, 2006, at 1:36 PM, Jim Weirich wrote:

--
_why: zenspider's most intense moments of solice are immediately following the slaughter [...]
_why: that topknot's the only thing keeping a lid on the righteous anger
bricolage: yeah, that and his flagrant obsession with dvorak

10

Ryan sez:

I do it all the time. The simple solution is to put a "test_dummy" in
the superclass. Really not hard and a fair tradeoff to have the right
code in the right place.

This has been suggested before to the Watir users. Many of them don't
like the fact that this results in an incorrect test case count.

I think the ideal solution would be to refactor the TestCase code
along
the lines of this patch, and thereby expose
TestCase.sorted_test_methods so that it could then be overlayed by
Modules.

I disagree. As I said before, we need less code, not more. (I do
wholly agree that test/unit needs refactoring, but to make it thinner
not thicker)

My suggestion is only to add a new seam, but i guess that amounts to a
bit of code. But only a bit.

Another reason that i packaged the code as a mod to TestCase directly
is that it makes it easier for people with large test suites to plug it
in. So it is more convenient for Watir users than a subclass.

However, I'm not quite sure whether you are saying
a) i (ryan) would prefer to do it a different way
b) i don't want to see this change made to the core test/unit code in
ruby
c) it is a bad idea for anyone to use this mod

I understand (a). I can see your reasons for (b), and actually have
been eager for feedback on whether this might be a good idea or not.
(although i did get some positive noises offline from Nathaniel Talbott
-- the test/unit author). i can see no grounds for (c) -- if this is
your position, please elaborate.

Bret

11

Exactly. Its hard to see how this fits into Test::Unit standing all alone like that.

···

On May 22, 2006, at 3:16 PM, Ryan Davis wrote:

On May 22, 2006, at 2:47 PM, Bret Pettichord wrote:

On 5/22/06, Eric Hodel <drbrain@segment7.net> wrote:

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

I don't see a patch there.

Click "Download Patch." Or click the attachments tab. I also just added a
link at the bottom of the page.

I think Eric's point was that it isn't a patch.

--
Eric Hodel - drbrain@segment7.net - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com

12

Ryan Davis wrote:

The problem with subclassing is that it doesn't combine well with
other
potential Test::Unit add-ins. (For that matter, I suspect the
patch way
doesn't combine well either, but I haven't looked at the details).

"potential add-ins" is a red flag for YAGNI.

For example, I have a Test::Unit::TestCase mod that allows FlexMock
objects to be automatically verified at the end of a test case. If I
make this a sub-class, then Brett can't use my FlexMock subclass with
his Watir subclass.

I guess I don't understand. Is your FlexMock mod a subclassed
testcase as well? If not, then where is the conflict?

Some background ... before last week, to use FlexMock in a test case,
you would say:

(I)
   def test_blah
     FlexMock.use("mockname") do |mock|
       mock.blah.blah.blah
       # test goes here
     end
   end

The block usage ensures that verification is run on the mock at the end
of the test. Although idiomatic Ruby, the block made (a) multiple mocks
awkward, and (b) the ability to refactor common mock creation/setup into
a separate method awkward.

Neither (a) nor (b) were show stoppers, just rather unsatisfying. What
I wanted to be able to write was:

(II)
   def test_blah
     mock = flexmock("mockname")
     mock.blah.blah.blah
     # test goes here
   end

Now multiple mocks and special mock creation methods are trivial. To
get to (II), I had to override teardown. I can use either a subclass or
module to do this. My choice was to use a module ...

However ...

Your suggestion to Brett was to use a subclass. If I had followed the
suggestion and used a subclass (rather than a module), then my add-in
and Brett's add-in could not interoperate. So, my counter suggestion to
Brett was to use a module as well (rather than a subclass).

Your extension seems to be orthogonal to
TestCase, and so belongs in a module to be mixed in wherever needed.
Brett's extension isn't orthogonal.

Ok, that's a possible reason for choosing a subclass over a module. I
hadn't seen the patch at the time of my original post, so I can't say if
it is orthognal or not. I shall ponder the issue :wink:

I want people to test. I want them to write all the unit, system,
integration, and performance tests they possibly want. I want them to
have the tools they need to do this, [...]

Absolute 100% agreement with the above.

*nod* I think this is a fine alternative. I'd probably take it a step
further and use it AND subclassing to make things both clearer and a
bit easier.

I would suggest (as a rule of thumb) that the library writers provide
modules and allow the end users to subclass and include the modules they
want/need.

-- Jim Weirich

···

On May 22, 2006, at 1:36 PM, Jim Weirich wrote:

--
Posted via http://www.ruby-forum.com/\.

13

Ryan sez:

I do it all the time. The simple solution is to put a "test_dummy" in
the superclass. Really not hard and a fair tradeoff to have the right
code in the right place.

This has been suggested before to the Watir users. Many of them don't
like the fact that this results in an incorrect test case count.

Quite frankly, I don't really care that they consider it to output an "incorrect" test case count. It works for many of us and is a fair tradeoff. I'd much rather have the requirement removed from test/unit outright, but I didn't design test/unit and I too disagree with requiring it (but! see below!). I however disagree wholly with making test/unit MORE complex under all but the most optimal of returns. I don't think your suggestion matches that requirement esp since it clearly violates the notion of a unit test.

That said, I think your proposal has merit, just not where you want it. I did just poke around a bit more and figgered something out that might be satisfactory for the both of us:

501 % ruby -rtest/unit -e 'class X < Test::Unit::TestCase; end'
Loaded suite -e
Started
F
Finished in 0.005603 seconds.

   1) Failure:
default_test(X) [-e:1]:
No tests were specified.

1 tests, 1 assertions, 1 failures, 0 errors
502 % ruby -rtest/unit -e 'class X < Test::Unit::TestCase; def default_test; end; end'
Loaded suite -e
Started
.
Finished in 0.00228 seconds.

1 tests, 0 assertions, 0 failures, 0 errors

YES, it still has an incorrect test count, but it also doesn't have as much "smell" as "test_dummy" in an abstract testcase.

I disagree. As I said before, we need less code, not more. (I do
wholly agree that test/unit needs refactoring, but to make it thinner
not thicker)

My suggestion is only to add a new seam, but i guess that amounts to a
bit of code. But only a bit.

Another reason that i packaged the code as a mod to TestCase directly
is that it makes it easier for people with large test suites to plug it
in. So it is more convenient for Watir users than a subclass.

How is it more convenient?

require 'monkeypatch'
class BlahTest < Test::Unit::TestCase
   execute :sequentially
   ...
end

vs

require 'systemtestcase'
class BlahTest < SystemTestCase
   execute :sequentially
   ...
end

Do you really see either of those as more convenient than the other?

However, I'm not quite sure whether you are saying
a) i (ryan) would prefer to do it a different way
b) i don't want to see this change made to the core test/unit code in
ruby
c) it is a bad idea for anyone to use this mod

I understand (a). I can see your reasons for (b), and actually have
been eager for feedback on whether this might be a good idea or not.
(although i did get some positive noises offline from Nathaniel Talbott
-- the test/unit author). i can see no grounds for (c) -- if this is
your position, please elaborate.

(a) obviously, but on the grounds of (b). I've purposefully not addressed (c) as I think its been addressed (several times) before, but I also agree that it is a bad idea to use the mod.

···

On May 22, 2006, at 11:58 AM, bpettichord@gmail.com wrote:

--
ryand-ruby@zenspider.com - Seattle.rb - Seattle.rb | Home
http://blog.zenspider.com/ - http://rubyforge.org/projects/ruby2c

14

Eric Hodel wrote:

···

On May 22, 2006, at 3:16 PM, Ryan Davis wrote:

On May 22, 2006, at 2:47 PM, Bret Pettichord wrote:

On 5/22/06, Eric Hodel <drbrain@segment7.net> wrote:

http://wiki.openqa.org/display/WTR/Test-Unit+Patch

I don't see a patch there.

Click "Download Patch." Or click the attachments tab. I also just
added a
link at the bottom of the page.

I think Eric's point was that it isn't a patch.

Exactly. Its hard to see how this fits into Test::Unit standing all
alone like that.

It's trivial to put that code into test/unit/testcase.rb, replacing the
lines of the existing self.suite method, around lines 40-60.

Cheers,
Dave

15

I was able to rewrite the code as a subclass. I was even able to
prevent the problem with the incorrect test count that had been a
reason i'd avoided this approach previously (see the initialize
method).

Comments please. I would be happy to present this code as a true patch
to the Test::Unit source if there is actually interest in seeing it in
that form. Is there? My sense was that most people felt this was best
delivered as a subclass. (I may actually pull out the ordering options
now, since they really don't serve a purpose -- yagni and all.)

Bret

require 'test/unit'

module Watir
    class TestCase < Test::Unit::TestCase
      @@order = :sequentially
      def initialize name
        throw :invalid_test if name == :default_test && self.class ==
Watir::TestCase
        super
      end
      class << self
        attr_accessor :test_methods, :order
        def test_methods
          @test_methods ||= []
        end
        def order
          @order || @@order
        end
        def default_order= order
          @@order = order
        end
        def sorted_test_methods
          case order
          when :alphabetically: test_methods.sort
          when :sequentially: test_methods
          when :reversed_sequentially: test_methods.reverse
          when :reversed_alphabetically: test_methods.sort.reverse
          else raise ArgumentError, "Execute option not supported:
#{@order}"
          end
        end
        def suite
          suite = Test::Unit::TestSuite.new(name)
          sorted_test_methods.each do |test|
            catch :invalid_test do
              suite << new(test)
            end
          end
          if (suite.empty?)
            catch :invalid_test do
              suite << new(:default_test)
            end
          end
          return suite
        end
        def method_added id
          name = id.id2name
          test_methods << name if name =~ /^test./
        end
        def execute order
          @order = order
        end
      end
    end
end