Testunit 0.1.6 problems

[Holden_Glova]

I agree with NaHi. I like having TestUnit fail when I have an empty TestCase
for the reason that NaHi has mentioned.

---

Signed,
Holden Glova

···

On Fri, 10 Jan 2003 00:05, NAKAMURA, Hiroshi wrote:

Hi,

From: “Bil Kleb” W.L.Kleb@larc.nasa.gov
Sent: Thursday, January 09, 2003 7:56 PM

[…] I’m not sure that’s a practice that ought to be supported. As an
XP’er, I’d say, “YAGNI!” and delete the TestCase as soon as it’s empty,
knowing that it’s cinchy to add back […]

[…] I’d like to hear some more on this first (if there are any other
opinions).

I like the failure when you have an empty TestCase for the XP YAGNI
reason you stated above.

+1 to failure, to keep health of test.
Empty testcase is exactly a bug of the testcase.

Regards,
// NaHi

[David_Alan_Black1 (David Alan Black)]

Hi –

Could there be a way to hook or override the default test, so
that one could relatively easily change the behavior?

Sure, it’s trivial, and that’s really at the core of the matter: where
do you introduce the overhead? Do you introduce it for those who want a
failure whenever they have an empty TestCase? Or, do you introduce it
for those who want to keep empty TestCases around? I think it has to be
one or the other - one set of users will have a bit of extra work, and
the other group of users will be able to start working out of the box.

To be honest, I still lean towards making the default to fail on an
empty TestCase. What’s a failure? It’s an invitation to implement.
What’s an empty TestCase? It’s an invitation to implement (as Dave
pointed out in ruby-talk:60964). If you want to silence the invitation,
you just add an empty test to match your empty test case.

I think what I had in mind was a little different. Let’s say every
empty test actually runs the #no_test method. That means that this:

def test_1
end

and this:

def test_1
no_test
end

are effectively the same.

The idea would then be: by overriding #no_test, you could change the
behavior, without having to change all the empty tests themselves.

David

···

On Fri, 10 Jan 2003 nathaniel@NOSPAMtalbott.ws wrote:

dblack@candle.superlink.net wrote:

–
David Alan Black
home: dblack@candle.superlink.net
work: blackdav@shu.edu
Web: http://pirate.shu.edu/~blackdav

[Martin_DeMello1 (Martin DeMello)]

nathaniel@nospamtalbott.ws wrote:

Sure, it’s trivial, and that’s really at the core of the matter: where
do you introduce the overhead? Do you introduce it for those who want a
failure whenever they have an empty TestCase? Or, do you introduce it
for those who want to keep empty TestCases around? I think it has to be
one or the other - one set of users will have a bit of extra work, and
the other group of users will be able to start working out of the box.

Wouldn’t enumerating them separately give you the best of both worlds?
Something like

2 failures, 3 empty tests

should keep everyone happy.

martin

[Tom_Clarke2 (Tom Clarke)]

They could be described as a separate category, a todo perhaps.
3 Errors, 5 Failures and 2 Todos.

-Tom

···

On Thu, Jan 09, 2003 at 11:08:26AM +0900, Hal E. Fulton wrote:

----- Original Message -----
From: “Mike Campbell” michael_s_campbell@yahoo.com
To: “ruby-talk ML” ruby-talk@ruby-lang.org
Sent: Wednesday, January 08, 2003 7:52 PM
Subject: Re: Test::Unit fails w/no tests [was: testunit 0.1.6 problems]

In my mind, there’s nothing whatsoever wrong with an empty unit test.

I tend to agree with Gavin that an empty unit test shouldn’t be flagged as
a
failure. I use them for placeholders too.

I can see that. But playing devil’s advocate, I think I see
the reasoning the other way also.

Aren’t the XPers always saying, “Write the test first. Since the code
doesn’t exist yet, the test will fail.” A new test, then, fails by
default. This was perhaps someone’s thinking. But I haven’t read this
whole thread.

I could live with it either way. I don’t test the way I should.

Hal

[Dmitri_Colebatch]

Hey all,

new to ruby and to the list, but this seems as good a thread as any to join
in with (o:

In my mind, there’s nothing whatsoever wrong with an empty unit test.

absolutely. imho testunit should resemble junit - two reasons why, I know
junit (o: but also (correct me if I’m wrong) junit is the de facto standard
in open source unit testing, yes?

I tend to agree with Gavin that an empty unit test shouldn’t be flagged
as
a failure. I use them for placeholders too.

I can see that. But playing devil’s advocate, I think I see
the reasoning the other way also.

Aren’t the XPers always saying, “Write the test first. Since the code
doesn’t exist yet, the test will fail.” A new test, then, fails by
default. This was perhaps someone’s thinking. But I haven’t read this
whole thread.

that argument then leads to “testing the test” - saying that

def test_foo
end

is a failing test is the same as saying that

def test_bar
    bar = Bar.new("bar")
end

fails because it should really be

def test_bar
    bar = Bar.new("bar")
    assert_equal("bar", b.bar, "bar not bar")
end

but like you say - its a moot point really, all tests should check
something.

cheers
dim

[NAKAMURA_Hiroshi1 (NAKAMURA, Hiroshi)]

Hi, Tim,

From: Tim Bates [mailto:tim@bates.id.au]
Sent: Thursday, January 09, 2003 8:22 PM

+1 to failure, to keep health of test.
Empty testcase is exactly a bug of the testcase.

Not necessarily. And not everyone practises XP as such.

I’m not a XPer. An old traditional test engineer.
To find a bug of the testcase, I vote to failure.
(Of cource, I can live with warning for it, but don’t
want to live with silence.) Not related to XP.

Regards,
// NaHi

[ts1 (ts)]

On a related note, I can't get this failure to happen - must be exclusive to
0.1.6 (I have 0.1.4) but I can't find the start of this thread. Can someone

yes, exclusive to 0.1.6

Does TC_Sub have this failure? If so, it will break my tests which is another
argument against it...

pigeon% ruby -rtest/unit
class TC_Super < Test::Unit::TestCase
        def testme
                assert_equal(2, 1+1, "Basic axioms of mathematics are wrong!")
        end
end

class TC_Sub < TC_Super
end
^D
Loaded suite -
Started
....
Finished in 0.002892 seconds.
2 tests, 2 assertions, 0 failures, 0 errors
pigeon%

pigeon% ruby -rtest/unit
class TC_Super < Test::Unit::TestCase
end

class TC_Sub < TC_Super
end
^D
Loaded suite -
Started
..
Failure!!!
run:
No tests were run.

Finished in 0.001873 seconds.
0 tests, 0 assertions, 1 failures, 0 errors
pigeon%

Guy Decoux

[Michael_Campbell1 (Michael Campbell)]

I can see that. But playing devil’s advocate, I think I see
the reasoning the other way also.

Aren’t the XPers always saying, “Write the test first. Since the
code
doesn’t exist yet, the test will fail.” A new test, then, fails by
default.

Only because there isn’t code TO test, not because of any “newness”
OF the test.

If I write a test that simply asserts “true”, the test is hardly
complete, but it doesn’t fail. To me, an empty unit test is like
that. Incomplete != failure.

···

=====

Yahoo IM: michael_s_campbell

---

Do you Yahoo!?
Yahoo! Mail Plus - Powerful. Affordable. Sign up now.

Yahoo Mail – organised email

[Dmitri_Colebatch]

Nathaniel,

as I understand the thread, because this has no assert
statements, it would be deemed a failure - correct?

No, it wouldn’t… the only thing that is (currently) an error is
running an empty TestSuite… as soon as you add an empty test, the
error will disappear.

thanks for the clarification.

cheers
dim

[Nathaniel_Talbott]

I’ll add that in my last Ruby project, I actually wrote my
own test running code so that each test class was run
individually (instead of all test methods being thrown into a
large soup and executed homogenously). The output was something like:

It’s on my TODO list to add something like Rubicon’s BulkTestRunner to
Test::Unit, which I believe will do something very similar to what
you’re talking about. I definitely think it’s very useful type of
runner.

Nathaniel

<:((><

···

Gavin Sinclair [mailto:gsinclair@soyabean.com.au] wrote:

RoleModel Software, Inc.
EQUIP VI

[Nathaniel_Talbott]

dblack@candle.superlink.net wrote:

I think what I had in mind was a little different. Let’s say
every empty test actually runs the #no_test method. That
means that this:

def test_1
end

and this:

def test_1
no_test
end

are effectively the same.

The idea would then be: by overriding #no_test, you could
change the behavior, without having to change all the empty
tests themselves.

Well, what’s being discussed is actually empty test suites, not empty
tests. So this is a failure:

def TC < Test::UnitTestCase
end

But this is not:

def TC < Test::Unit::TestCase
def test_1
end
end

That said, I do think I can go the direction you’re talking about and
make it so that one little tweak to the framework will change this
globally, as opposed to having to silence things on an individual basis.
Perhaps:

require ‘test/unit/testcase’

module Test
module Unit
class TestCase
def default_test
end
end
end
end

You could throw something like that in a file and require it in your
tests to silence the warning. Or, if you wanted to, you could do a check
of your own instead.

Anyhow, that’s what I’m currently thinking. It is, of course, subject to
change when I actually get around to implementing it

Nathaniel

<:((><

···

RoleModel Software, Inc.
EQUIP VI

[Nathaniel_Talbott]

But it’s distracting you from real failures as well.

Only if it’s not a real failure itself. My plan now is to implement it
in terms of a regular assertion, so that it is, indeed, a real boy, er,
failure

As I just wrote to David, you’d be able to override the behavior very
easily on a global basis, so if you didn’t want a real failure for an
empty, you could just require a file and not get it. I hope that’s a
good compromise.

My main issue is that (no test → failure) is not
theoretically sound. A failure, contrary to what I said
before, is an assertion that, well, fails. (I said it was an
uncaught exception, which is an error.)

So if there is no test, and therefore no assertion, there is
no failure.

So we just say that there is a default test that runs a default
assertion, and that that assertion is by default a #flunk. Does that
make it a bit more theoretically sound? Although I can see that some
would not think there should be a default test at all, which is where
we’ll just have to agree to disagree.

This is all pretty high-falutin’ theory.

Yup, and I’m lovin’ it! I enjoy having a good theoretical discussion,
especially when it feeds in to something practical I need to do

But I think
software should be theoretically sound, especially
frameworks. We all know that dodgy hacks can come back to
haunt us. Well, dodging the theory is a small step down that road.

Agreed on all points. I think matz is a great example of being careful
when tweaking a framework, and I’m doing my best to follow his example.
Everyone who’s commented here has helped me out a lot. While some still
might not like it 100%, I think the direction I’m planning on taking now
is a_lot more sound than what I was planning on doing originally.

Sorry for harping on about it. It’s good that lots of people
have chimed in. I’ll leave it alone, now, and won’t complain
about whatever decision you make.

Hey, thanks for harping! I hope you do the same for any other things
that come up.

Nathaniel

<:((><

···

Gavin Sinclair [mailto:gsinclair@soyabean.com.au] wrote:

RoleModel Software, Inc.
EQUIP VI

[Tim_Bates]

Good, that was exactly what I expected, given my understanding of Ruby and
Test::Unit.

Tim Bates

···

On Thu, 9 Jan 2003 10:22 pm, ts wrote:

2 tests, 2 assertions, 0 failures, 0 errors

–
tim@bates.id.au

[Nathaniel_Talbott]

They could be described as a separate category, a todo perhaps.
3 Errors, 5 Failures and 2 Todos.

But to play devil’s advocate, isn’t a failure by definition a todo?

Nathaniel

<:((><

···

Tom Clarke [mailto:tom@u2i.com] wrote:

RoleModel Software, Inc.
EQUIP VI

[Martin_Hart]

On 1/10/03, 3:51:57 PM, nathaniel@NOSPAMtalbott.ws wrote regarding Re:
Test::Unit fails w/no tests [was: testunit 0.1.6 problems]:

But it’s distracting you from real failures as well.

Only if it’s not a real failure itself. My plan now is to implement it
in terms of a regular assertion, so that it is, indeed, a real boy, er,
failure

Excuse me for butting in late on in the thread, this may not be of any
relevance/importance, but have you considered “feature” compatability
with other versions of xUnit?

While it is a minor point, I do wonder if Test::Unit should behave the
same way as all the other tools based on xUnit?

Either way - it’s a great module and I’m not going to stop using it if
this feature stays in

Cheers,
Martin

···

Gavin Sinclair [mailto:gsinclair@soyabean.com.au] wrote:

–
Martin Hart Tel: +44 (0) 1582 618468
Arnclan Fax: +44 (0) 1582 619596
Union Street, E-mail: martin@zsdfherg.com
Dunstable, Beds LU6 1EX

[Gavin_Sinclair]

I’m happy with that. It’s not the default behaviour I would prefer,
but it’s well thought out, it’s overridable within the framework, and
it’s theoretically sound

Gavin

···

On Saturday, January 11, 2003, 2:51:57 AM, nathaniel wrote:

My main issue is that (no test → failure) is not
theoretically sound. A failure, contrary to what I said
before, is an assertion that, well, fails. (I said it was an
uncaught exception, which is an error.)

So if there is no test, and therefore no assertion, there is
no failure.

So we just say that there is a default test that runs a default
assertion, and that that assertion is by default a #flunk. Does that
make it a bit more theoretically sound? Although I can see that some
would not think there should be a default test at all, which is where
we’ll just have to agree to disagree.

[Dmitri_Colebatch]

I would see the difference between failures (and errors) and TODOs is that
I’d commit a TODO into cvs, but wouldn’t commit a failing test into cvs.

my 2c

cheers
dim

···

Tom Clarke [mailto:tom@u2i.com] wrote:

They could be described as a separate category, a todo perhaps.
3 Errors, 5 Failures and 2 Todos.

But to play devil’s advocate, isn’t a failure by definition a todo?

[Tom_Clarke2 (Tom Clarke)]

True, but I would fix all my failing tests before implementing new
ones.

Or to put it another way, a failing test is a reminder to implement
functionality as specified by the failing test. An empty test is a
reminder to specify some functionality.

Or yet another way - I would be much happier going to lunch with some
functionality left to specify than functionality broken.

So yes, both are todos but it may be useful to separate the two.

I’m probably thinking far too much about this ;-).

-Tom

···

On Fri, Jan 10, 2003 at 01:03:44PM +0900, nathaniel@NOSPAMtalbott.ws wrote:

Tom Clarke [mailto:tom@u2i.com] wrote:

They could be described as a separate category, a todo perhaps.
3 Errors, 5 Failures and 2 Todos.

But to play devil’s advocate, isn’t a failure by definition a todo?

[Nathaniel_Talbott]

Martin Hart said:

Excuse me for butting in late on in the thread, this may not be of any
relevance/importance, but have you considered “feature” compatability
with other versions of xUnit?

While it is a minor point, I do wonder if Test::Unit should behave the
same way as all the other tools based on xUnit?

While I want to make the transition for users of other xUnits as painless
as possible, I hope that a testing framework written in Ruby can far
surpass one written in C++ or Java. Plus, if you compare the original
xUnit (sUnit, or Smalltalk Unit) to something like JUnit, you’ll see that
there’s actually a lot of variance already.

While I don’t ever want to break compatibility/similiarity without good
reason, I’m always looking for good reasons to do exactly that.

Either way - it’s a great module and I’m not going to stop using it if
this feature stays in

Thanks - I’m glad you like it.

Nathaniel