Unit testing is considered harmful

ruby-talk
1

I’m sorry, I could help it. I just find this considered harmful thing
completely overused.
Anyway todays topic is about that narrowminded focus on Unit Tests.

We just saw the programming contest having the test program tested, rather
than the applications. Of course UnitTest never claimed to be complete.

I read an interesting article about random testing (what and where is
forgotten), but the conclusion was that just hammering random data into a
system would give you less coverage than well designed tests, but when
counting in the time / effort spent to build these tests, you would get more
coverage by brute force.

This is of course awfully general, and UnitTesting is to be taken seriously.
But adding randomness as a key feature to a test suite is a good thing.

Another good thing, where applicable, is to specify the invariants that must
hold true. This can relatively easily be specified.

The invariant test is applied to a input/output dataset, object hierarchy,
datastruture or whatever. The random test suit then generates as many
testcases as possible and use the invariant to validate the results.
If you were clever in the random generator, you know now exactly what random
seed caused the error, and you can debug through that particular case and
see what invariant failed.

An example is a tree datastructure where each node has a cached accumulated
value of the child nodes data. The invariant will say something about how
the cache relates to actual data in child nodes. If there is a bug in the
cache update logic, the invariant will find it and have an exact test case
to reproduce.

I have in fact been using this method recently, in it pays itself back 100
fold in a complex datastructure. I know because I’ve tried manual debugging
of a similar problem.

If we take programming contest as example, it is important to identify the
nature of the problem. I suggested a relation to the assingment problem,
which may or may not be appropriate. But that view gives a good idea of
which data sets are valid and which a not, instead of extrapolating from a
few unit test cases. This view can then be used to build the verification
function. Having this function is not sufficient to ensure that submissions
are correct. We need to feed the contenders with test data that can be
verified.

The upcoming ICFP programming contest will heavily test contenders with
similar techniques, although the probably predefine a large test data set.
Some of the most successful solutions are not the most clever ones, but
those that have good measures for testing both during development, but
certainly also during runtime. The spend good CPU cycles on rejecting
optimizations that are incorrect due to some undetected bug.

Mikkel

2

I’m sorry, I could help it. I just find this considered harmful thing
completely overused.
Anyway todays topic is about that narrowminded focus on Unit Tests.

We just saw the programming contest having the test program tested, rather
than the applications. Of course UnitTest never claimed to be complete.

Coincidentally, I submitted a question about this–CodingChallenge0002
vs. unit tests–to the extreme-programming mailing list this morning.
I was commenting on my surprise that we kept adding unit tests after
the fact, and breaking existing solutions. I referred people to the
wiki page. What happened is people replied back with detailed
questions, specifically stating that the requirements were too vague
and asking for clarification. . . . Which is just what happened here on
ruby-talk when the challenge was first announced. [[Not pointing
fingers!! I enjoyed the challenge and learned a lot from it.]]

[…]

If we take programming contest as example, it is important to identify the
nature of the problem. I suggested a relation to the assingment problem,
which may or may not be appropriate. But that view gives a good idea of
which data sets are valid and which a not, instead of extrapolating from a
few unit test cases.

I agree. In no way when I approached the problem, was I extrapolating
from what unit test cases were present. I had asked David on the list
for clarifications to a couple questions regarding duplicates and whether
the items could be considered to be sortable. I also read David’s reply
to someone else where he described thinking about the problem as drawing
lines, connecting elements in the 1st array to rows in the second. . .
My point is two-fold: one, that one or more of us should probably have
considered going back to the wiki page and updating the requirements to
include the new information/clarifications expressed on ruby-talk; and
two, that I–at the time–had no problem adding further unit tests as
we went along, thinking of it as codifying what was already clear to
me in the requirements (forgetting, that the requirements were in
existence now partially on the wiki page, but partially in ruby-talk
messages…)

So I was somewhat at a loss as to how new unit tests were breaking so
many of the existing solutions… But I think, as was brought to my
attention on the XP list, that the essence of the problem may have
lied in the incomplete definition of the requirements on the wiki page,
and . . . . well. Pretty much that. :slight_smile:

However, I don’t, myself, see how this could be construed as a failure
of the unit tests… Unless it was thought that the unit test cases
were to substitute as implicit definitions of the requirements. I
would agree that we may have witnessed that approach being harmful.
But I still don’t blame the unit tests themselves…

Just my sqrt(4) cents…

Regards,

Bill

···

From: “MikkelFJ” mikkelfj-anti-spam@bigfoot.com

3

“MikkelFJ” mikkelfj-anti-spam@bigfoot.com wrote in message
news:3d5457d0$0$43947$edfadb0f@dspool01.news.tele.dk…

I’m sorry, I could help it. I just find this considered harmful thing
completely overused.

I suspect that Mikkel worded the opening line for his post this way
intentionally, but an aside:

As many readers of comp.lang.ruby and ruby-talk will already know, Edsger W.
Dijkstra, author of the famous “Go To Statement Considered Harmful” letter
(http://www.acm.org/classics/oct95), passed away on August 6. Here’s a link
to one obituary (from the UTex CS department):

http://www.cs.utexas.edu/users/UTCS/notices/dijkstra/ewdobit.html

Dijkstra’s contributions to the field of computer science are overwhelming;
to quote the obit: “Dijkstra enriched the language of computing with many
concepts and phrases, such as structured programming, separation of
concerns, synchronization, deadly embrace, dining philosophers, weakest
precondition, guarded command, the excluded miracle, and the famous
‘semaphores’ for controlling computer processes. The Oxford English
Dictionary cites his use of the words ‘vector’ and ‘stack’ in a computing
context.”

Lyle

4

Hi –

From: “MikkelFJ” mikkelfj-anti-spam@bigfoot.com

I’m sorry, I could help it. I just find this considered harmful thing
completely overused.
Anyway todays topic is about that narrowminded focus on Unit Tests.

We just saw the programming contest having the test program tested, rather
than the applications. Of course UnitTest never claimed to be complete.

I can’t suppress a smile at the idea of my being used as an example of
excessive fanaticism about unit testing :slight_smile:

Coincidentally, I submitted a question about this–CodingChallenge0002
vs. unit tests–to the extreme-programming mailing list this morning.
I was commenting on my surprise that we kept adding unit tests after
the fact, and breaking existing solutions. I referred people to the
wiki page. What happened is people replied back with detailed
questions, specifically stating that the requirements were too vague
and asking for clarification. . . . Which is just what happened here on
ruby-talk when the challenge was first announced. [[Not pointing
fingers!! I enjoyed the challenge and learned a lot from it.]]

I guess ideally this kind of thing would be either on a wiki or on a
mailing list, but not both. It’s probably the fault of my quirky
habits: I liked the idea of putting the problem and the solutions on
the wiki, but for whatever reason I tend to be more at home with
email-based than wiki-based discussion. Oh, and people did ask
questions here on the list :slight_smile:

So I was somewhat at a loss as to how new unit tests were breaking so
many of the existing solutions… But I think, as was brought to my
attention on the XP list, that the essence of the problem may have
lied in the incomplete definition of the requirements on the wiki page,
and . . . . well. Pretty much that. :slight_smile:

Of course, I understood the requirements, and yet the original
non-airtight bunch of tests was written by me :slight_smile: That’s not too
suprising, though I’ve found it quite interesting to see how elusive a
really “complete” set of tests for this problem (if there can be such
a thing) turns out to be.

However, I don’t, myself, see how this could be construed as a failure
of the unit tests… Unless it was thought that the unit test cases
were to substitute as implicit definitions of the requirements. I
would agree that we may have witnessed that approach being harmful.
But I still don’t blame the unit tests themselves…

Ever since last year’s Ruby Conference, where we talked a lot about
unit testing, I’ve been intrigued by the idea of having a test suite
serve as an actual definition of an interface – and even different
test suites defining different versions. In the case of this coding
challenge, however, I was aware that the tests probably wouldn’t be
enough – which was on my mind when I wrote:

The challenge is to implement the method so that the tests pass and
the requirements are met.

:slight_smile:

It certainly has been an interesting little case study.

David

···

On Sat, 10 Aug 2002, Bill Kelly wrote:

–
David Alan Black
home: dblack@candle.superlink.net
work: blackdav@shu.edu
Web: http://pirate.shu.edu/~blackdav

5

As many readers of comp.lang.ruby and ruby-talk will already know, Edsger
W.
Dijkstra, author of the famous “Go To Statement Considered Harmful” letter
(http://www.acm.org/classics/oct95), passed away on August 6. Here’s a
link
to one obituary (from the UTex CS department):

http://www.cs.utexas.edu/users/UTCS/notices/dijkstra/ewdobit.html

Dijkstra’s contributions to the field of computer science are
overwhelming;
to quote the obit: “Dijkstra enriched the language of computing with many
concepts and phrases, such as structured programming, separation of
concerns, synchronization, deadly embrace, dining philosophers, weakest
precondition, guarded command, the excluded miracle, and the famous
‘semaphores’ for controlling computer processes. The Oxford English
Dictionary cites his use of the words ‘vector’ and ‘stack’ in a computing
context.”

I didn’t know that. That is so sad.

I live ten miles from UT, and I didn’t know. Obviously I am
far out of the loop.

Hal

···

----- Original Message -----
From: “Lyle Johnson” lyle@knology.net
Newsgroups: comp.lang.ruby
To: “ruby-talk ML” ruby-talk@ruby-lang.org
Sent: Saturday, August 10, 2002 9:26 AM
Subject: Re: Unit testing is considered harmful

6

“Lyle Johnson” lyle@knology.net wrote in message
news:ula8ff2batcv8c@corp.supernews.com…

“MikkelFJ” mikkelfj-anti-spam@bigfoot.com wrote in message
news:3d5457d0$0$43947$edfadb0f@dspool01.news.tele.dk…

I’m sorry, I could help it. I just find this considered harmful thing
completely overused.

I suspect that Mikkel worded the opening line for his post this way
intentionally, but an aside:

I actually I meant couldn’t - but of course I could help it. The ironey of
considered harmful was intententional:
If UnitTesting is fanatic, so is the the Considered Harmful movement.
I’m not sure about this, but I believe the original Goto considered harmful
did indeed accept that Goto were useful in some places but not as the
primary means of control (as we have seen it in early Basic). This then got
elevated into all Goto is bad, much like the disciples of Socrates digressed
and made a point of counterarguing everything.

To the extreme there was recently a letter from Peter Coad about “Lists
Considered Harmful” stating the bleeding obvious that trees are better than
lists for indexing large data sets. (When do I get to be a Guru ™ :wink: )

I do use Unit Testing myself (actually I have written a much easier to use,
portable, downscaled C++ framework if anyone is interested). But again this
goes against these hugely bloated testing frameworks that will make you code
difficult to port and make it less future proof.

The real point is automated regression test - and I do believe the random
test generations is an important tool here. It also works well with design
by contract because the random testcases triggers the contract invariants.

As many readers of comp.lang.ruby and ruby-talk will already know, Edsger
W.
Dijkstra, author of the famous “Go To Statement Considered Harmful” letter
(http://www.acm.org/classics/oct95), passed away on August 6. Here’s a
link
to one obituary (from the UTex CS department):

I was about to say that is very sad - but then that is the way we all go
eventually. Dijkstras shortest path algorithm is one of may favorite
algorithms. It is a very elegant way to efficiently solve a seemingly
difficult problem.

Mikkel

7

I still have his book “a discipline of programming” here at my desk. I never
expected everyone else to age with me.

8

Hi –

Coincidentally, I submitted a question about this–CodingChallenge0002
vs. unit tests–to the extreme-programming mailing list this morning.
I was commenting on my surprise that we kept adding unit tests after
the fact, and breaking existing solutions. I referred people to the
wiki page. What happened is people replied back with detailed
questions, specifically stating that the requirements were too vague
and asking for clarification. . . . Which is just what happened here on
ruby-talk when the challenge was first announced. [[Not pointing
fingers!! I enjoyed the challenge and learned a lot from it.]]

I guess ideally this kind of thing would be either on a wiki or on a
mailing list, but not both. It’s probably the fault of my quirky
habits: I liked the idea of putting the problem and the solutions on
the wiki, but for whatever reason I tend to be more at home with
email-based than wiki-based discussion. Oh, and people did ask
questions here on the list :slight_smile:

Spreading the requirements around is a problem, but in these circumstances
completely expected. The reality is that if this were a serious problem facing
a paid developer, they would surely pinpoint requirements. It was a difficult
problem to viaualise (in computer language, anyway; I’ve never had any problem
with the mathematics of 1-1 correspondance), so misunderstandings were rife,
and in short it wasn’t a great experiment if one’s aim was to test unit
testing.

It certainly has been an interesting little case study.

Yes, I wish I had the time and sufficient understanding to submit an entry.
When’s the next one?

David

Comment on unit tests in general: you have to - to some extent - prove your
program correct, and put lots of sanity checks in the code (i.e. invariants);
but unit tests are really useful for just being able to repeat manual tests.
Not an insightful comment, but so long as you don’t abuse them by relying on
them too much, they’re great!

Oh, another comment. Many applications are a lot more “practical” than the
textbook examples we always hear (and was repeated here with balancing trees or
some such). These apps are IO-intensive, specific-calculation-intensive, or
whatever. They don’t usually fit the mould of rigorous requirements analysis,
proofs and invariants. How I wish a book were written that broke this mould.
Suggestion, anyone?

Gavin

···

----- Original Message -----
From: dblack@candle.superlink.net

9

Not being “schooled” in unit test development, but contributing several
infrastructure pieces to projects recently (FreeRIDE/FreeBASE, etc). I
wrote this particularly interesting part of the FreeRIDE infrastructure
named the ‘databus’ and I found myself iterating toward complete unit
testing of my public interface/behavior while the unit tests served as a
“sounding board” for my design (API simplicity, etc) as well as a
verifier of functionality. It was much more back and forth for me than
test first…all specifications defined…write code.

More recently I had to make a major change to the code (like someone was
depending on me getting it done as fast as possible). A little global
search/replace covered most things…refactored some class names, then
my tests failed…which I updated until they passed (to the newly
refactored interface). I was moving quick…which was why I jumped into
the code first…but the unit tests reminded me of what I could have
done wrong. After 5 minutes I checked the code back in to CVS feeling
very comfortable that my changes were solid and tested…which they were
;-).

-rich

···

-----Original Message-----
From: dblack@candle.superlink.net [mailto:dblack@candle.superlink.net]
Sent: Saturday, August 10, 2002 1:08 AM
To: ruby-talk ML
Subject: Re: Unit testing is considered harmful

Ever since last year’s Ruby Conference, where we talked a lot about
unit testing, I’ve been intrigued by the idea of having a test suite
serve as an actual definition of an interface – and even different
test suites defining different versions. In the case of this coding
challenge, however, I was aware that the tests probably wouldn’t be
enough – which was on my mind when I wrote:

10

This is actually a very important part (or consequence) of the “test
first” strategy: the unit tests not only serve to make sure that what
you have works, but let you concentrate first on the interface for
whatever it is you’re trying to build.

I am always having the experience of thinking I wanted to write
something in a certain way, sitting down and writing unit tests,
and just while writing the tests realizing that I need a different
API than what I’d thought.

cjs

···

On Sat, 10 Aug 2002, Rich Kilmer wrote:

…and I found myself iterating toward complete unit
testing of my public interface/behavior while the unit tests served as a
“sounding board” for my design (API simplicity, etc) as well as a
verifier of functionality. It was much more back and forth for me than
test first…all specifications defined…write code.

–
Curt Sampson cjs@cynic.net +81 90 7737 2974 http://www.netbsd.org
Don’t you know, in this new Dark Age, we’re all light. --XTC