Test::Unit sequencing

A question for more experienced Test::Unit users.

I want to do a fairly complex sequence of tests which carries state between
them: e.g. create a blank database, call some methods which should create
some entries, then test they are there, call some other methods which should
find and update those entries, test they have been changed, and so on.

I can use ‘setup’ to create the empty environment, but how do I make sure
all the other tests are performed in the right sequence? Should I just write
this all as one big test method?

Cheers,

Brian.

This thread should interest you: [ruby-talk:62836]

···

On Thu, May 15, 2003 at 05:25:45PM +0900, Brian Candler wrote:

A question for more experienced Test::Unit users.

I want to do a fairly complex sequence of tests which carries state between
them: e.g. create a blank database, call some methods which should create
some entries, then test they are there, call some other methods which should
find and update those entries, test they have been changed, and so on.

I can use ‘setup’ to create the empty environment, but how do I make sure
all the other tests are performed in the right sequence? Should I just write
this all as one big test method?


_ _

__ __ | | ___ _ __ ___ __ _ _ __
’_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |
__ \ | | | | | (| | | | |
.__/ _,
|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

‘Mounten’ wird für drei Dinge benutzt: ‘Aufsitzen’ auf Pferde, 'einklinken’
von Festplatten in Dateisysteme, und, nun, ‘besteigen’ beim Sex.
– Christa Keil

Hi Brian,

“The first rule of unit testing is, don’t let tests depend on each
other.
The second rule of unit testing is, don’t let tests depend on each
other.”
:wink:

You could write them in one big method. But if it is at all possible,
try to create some kind of mock-object that fakes the behaviour of the
environment. (Even if this involves changing the design of the
application or the environment it is probably worth it).

The one-big-test way can work nicely too to find bugs, but it’s not
really unit testing, so that’s why the unit testing tools won’t really
fit the problem.

/Anders

···

— Brian Candler B.Candler@pobox.com wrote:

I can use ‘setup’ to create the empty environment, but how do I make
sure
all the other tests are performed in the right sequence? Should I
just write
this all as one big test method?

=====


Anders Bengtsson ndrsbngtssn@yahoo.se
Stockholm, Sweden


Gå före i kön och få din sajt värderad på nolltid med Yahoo! Express
Se mer på: http://se.docs.yahoo.com/info/express/help/index.html

Thank you, very interesting. Conclusion: write one test method, and divide
it up into subtest methods which are called explicitly in the right order.

Cheers,

Brian.

(Boggles at the fact that there have been 11,239 posts to this list so
far this year… the first was ruby-talk:60251)

···

On Thu, May 15, 2003 at 05:29:35PM +0900, Mauricio Fern?ndez wrote:

This thread should interest you: [ruby-talk:62836]

Hi Brian,

“The first rule of unit testing is, don’t let tests depend on each
other.
The second rule of unit testing is, don’t let tests depend on each
other.”
:wink:

i know this creed, but it seems not to apply where databases are concerned…

You could write them in one big method. But if it is at all possible, try to
create some kind of mock-object that fakes the behaviour of the environment.
(Even if this involves changing the design of the application or the
environment it is probably worth it).

IMHO. in brian’s (and some of my) situation the database is the mock object
carying the state - i hardly see the difference between calling a(), b(), c()
myself from within test_foo() vs simply making

a() -> test_a()
b() -> test_b()
c() -> test_c()

isn’t this just semantics? i mean, where do you draw the line, should
assertions also be order independent? i guess it seems that tests of systems
which carry state cannot avoid order dependancies - it’s simply a question of
where you enforce it. a while back one of the Test::Unit developers told me
they ran the tests in alphabetical order for exactly this reason - so i took
advantage of it.

-a

···

On Thu, 15 May 2003, [iso-8859-1] Anders Bengtsson wrote:

The one-big-test way can work nicely too to find bugs, but it’s not really
unit testing, so that’s why the unit testing tools won’t really fit the
problem.

/Anders

=====


Anders Bengtsson ndrsbngtssn@yahoo.se
Stockholm, Sweden


Gå före i kön och få din sajt värderad på nolltid med Yahoo! Express
Se mer på: http://se.docs.yahoo.com/info/express/help/index.html

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
====================================

Considering I’ve posted 336 messages (not including this one) to ruby-talk
so far in 2003, I’m responsible for around 3% of the volume, with almost
2.5 postings per day :slight_smile: Now, who’d have the questionable honor of being
the fastest poster?.. (could as well be me :-P)

batsman@tux-chan:/tmp/milio$ expand -t2 analize.rb

require “parsedate”

count = 0

state = :FIND_MSG
date = to = nil

while line = gets
case state
when :FIND_MSG
state = :FIND_DATA if line =~ /^From /
when :FIND_DATA
if line =~ /^To: (.)/ && !to
to = $1
elsif line =~ /^Date: (.
)/ && !date
date = $1
end
if to && date
year, month = ParseDate.parsedate date
count += 1 if to == "ruby-talk@ruby-lang.org" && year == 2003
to = date = nil
state = :FIND_MSG
end
end
end

puts "Sent #{count} messages to ruby-talk this year."
batsman@tux-chan:/tmp/milio$ cat Sent.all | ruby analize.rb
Sent 336 messages to ruby-talk this year.

···

On Thu, May 15, 2003 at 06:34:50PM +0900, Brian Candler wrote:

On Thu, May 15, 2003 at 05:29:35PM +0900, Mauricio Fern?ndez wrote:

This thread should interest you: [ruby-talk:62836]

Thank you, very interesting. Conclusion: write one test method, and divide
it up into subtest methods which are called explicitly in the right order.

Cheers,

Brian.

(Boggles at the fact that there have been 11,239 posts to this list so
far this year… the first was ruby-talk:60251)


_ _

__ __ | | ___ _ __ ___ __ _ _ __
’_ \ / | __/ __| '_ _ \ / ` | ’ \
) | (| | |
__ \ | | | | | (| | | | |
.__/ _,
|_|/| || ||_,|| |_|
Running Debian GNU/Linux Sid (unstable)
batsman dot geo at yahoo dot com

The memory management on the PowerPC can be used to frighten small children.
– Linus Torvalds

I think I can see where Anders is coming from: I could replace my database
(the bit which carries state) with a mock DBI server, and check that the
main code requests the correct sequence of SQL statements (i.e. assert on
the work which the black box is doing, not just on the results coming out of
it). But this is a lot of work - as well as checking that the correct
requests hit the mock object, I have to supply all the correct responses.

Actually, I am doing some state checks (i.e. after doing an operation
which should insert something into the database, I do a direct SQL select
from it and check what I expect to be there is), so perhaps my methodology
isn’t completely consistent; but this lets me be selective about the things
that I check, picking a few pertinent rows and fields here and there to
check to get sufficient coverage, with minimal work.

You can say this is not pure unit testing, but from a pragmatic point of
view it seems to be getting the work done nicely right now :slight_smile:

Cheers,

Brian.

···

On Fri, May 16, 2003 at 12:34:48AM +0900, ahoward wrote:

IMHO. in brian’s (and some of my) situation the database is the mock object
carying the state - i hardly see the difference between calling a(), b(), c()
myself from within test_foo() vs simply making

a() -> test_a()
b() -> test_b()
c() -> test_c()

isn’t this just semantics? i mean, where do you draw the line, should
assertions also be order independent? i guess it seems that tests of systems
which carry state cannot avoid order dependancies - it’s simply a question of
where you enforce it.

sure sounds like alot - does anyone know of a means to compare this with other
comp.lang groups like perl, python, c, java, c++?

would be interesting to see.

-a

···

On Thu, 15 May 2003, Brian Candler wrote:

(Boggles at the fact that there have been 11,239 posts to this list so
far this year… the first was ruby-talk:60251)

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
====================================

ahoward wrote:

“The first rule of unit testing is, don’t let tests depend on each
other.
The second rule of unit testing is, don’t let tests depend on each
other.”
:wink:

i know this creed, but it seems not to apply where databases are concerned…

It applies particulary in the case of databases. :slight_smile: Writing simple unit
tests requires that you decouple your logic from your database code,
which is usually considered a good thing.

IMHO. in brian’s (and some of my) situation the database is the mock object
carying the state
[snip]

Well, kinda, but couldn’t you replace this slow and complex mock object
with a fast simple one, like something that just verifies SQL against a
known-good list and returns predefined results?

···

A n d e r s B e n g t s s o n | ndrsbngtssn@yahoo.se
Stockholm, Sweden |

In article Pine.LNX.4.53.0305151810520.12382@eli.fsl.noaa.gov,

(Boggles at the fact that there have been 11,239 posts to this list so
far this year… the first was ruby-talk:60251)

sure sounds like alot - does anyone know of a means to compare this with other
comp.lang groups like perl, python, c, java, c++?

would be interesting to see.

One way is to ask groups.google.com’s advanced search - I asked for
all the posts in comp.languages.perl.* between 1 Jan 2003 and today and
the stats said:

Searched Groups for group:comp.lang.perl.* from 1 Jan 2003 to today.
Results 1 - 10 of about 13,500. Search took 0.39 seconds.

but each result is a thread, and I’m too lazy to add up the articles
:slight_smile: It clams comp.lang.ruby has ~ 8260 threads for the period.

Whether that comes close enough to answering your question I don’t know;
I know that threads in comp.lang.perl.misc used to run and run so their
average thread length could be bigger than c.l.r’s

Hope this helps,

Mike

···

ahoward ahoward@fsl.noaa.gov wrote:

On Thu, 15 May 2003, Brian Candler wrote:

mike@stok.co.uk | The “`Stok’ disclaimers” apply.
http://www.stok.co.uk/~mike/ | GPG PGP Key 1024D/059913DA
mike@exegenix.com | Fingerprint 0570 71CD 6790 7C28 3D60
http://www.exegenix.com/ | 75D2 9EC4 C1C0 0599 13DA

It applies particulary in the case of databases. :slight_smile: Writing simple unit
tests requires that you decouple your logic from your database code, which
is usually considered a good thing.

my logic is very tightly coupled with the database code, in fact it is a
database: a bi-temporal database emulator to be precise - postgresql wrapped
in a ruby api:

http://raa.ruby-lang.org/list.rhtml?name=btpgsql

Well, kinda, but couldn’t you replace this slow and complex mock object with
a fast simple one, like something that just verifies SQL against a
known-good list and returns predefined results?

no. it’s the interaction between the database and my api which is being
tested! it is slow and very complex, much more so because of the
bi-temporal aspect, but this slow complex behavior is what i’m testing.

at the time - i couldn’t see any alternatives (of course there probably are).

-a

···

On Fri, 16 May 2003, Anders Bengtsson wrote:

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
====================================

I don’t think I agree with the statement that decoupling logic from
database code is a good thing. I’m a little uncertain on my views as to
the statement because I don’t understand what it means to decouple
logic from database code, and I am not sure I understand what Mr.
Bengtsson means by “database code”.

···

On Thursday, May 15, 2003, at 04:51 PM, Anders Bengtsson wrote:

ahoward wrote:

“The first rule of unit testing is, don’t let tests depend on each
other.
The second rule of unit testing is, don’t let tests depend on each
other.”
:wink:

i know this creed, but it seems not to apply where databases are
concerned…

It applies particulary in the case of databases. :slight_smile: Writing simple unit
tests requires that you decouple your logic from your database code,
which is usually considered a good thing.

ahoward ahoward@fsl.noaa.gov wrote in message news:Pine.LNX.4.53.0305152132400.12382@eli.fsl.noaa.gov

no. it’s the interaction between the database and my api which is being
tested! it is slow and very complex, much more so because of the
bi-temporal aspect, but this slow complex behavior is what i’m testing.

at the time - i couldn’t see any alternatives (of course there probably are).

Well, it sounds like the solution you came upon – having long tests
that have lots of little indirect asserts, called in a specific order
– works, more or less. Unit tests ideally shouldn’t be really long –
they’re less useful as documentation that way – but then sometimes
you can’t help it. I find that when I’m testing high-level objects I
tend to write pretty long, involved unit-tests. When I’m testing
low-level stuff I like my tests to be just a few lines.

I’ve never written any code that’s so low-level, and thus would
require so much setup & teardown time, but I suppose if I were
eventually I’d look into having a separate machine for continuous
integration. ThoughtWorks’ Cruise Control, for example, continually
pings the CVS server, waiting for checkin to calm down, and then
checks it out and runs full tests. One of its goals, specifically, is
to make it easier to have tests running even if the whole suite takes
15 minutes to run.

Of course, maybe you don’t need that yet. Just something to think
about in the future.

Francis

By database code I meant the parts of the code that knows about the
database. If your application is written in a mostly OO-style you’ll
probably have the “business logic” of the app separated from the gritty
details about how to store a value in a database column.
But then again, most small apps that use a database are built very much
around the database, without that separation. It makes them easier to
develop, but less object oriented, which makes them much harder to unit
test. (But other kinds of testing works ok).

/Anders

···

— Mark Wilson mwilson13@cox.net wrote:

I don’t think I agree with the statement that decoupling logic from
database code is a good thing. I’m a little uncertain on my views as
to
the statement because I don’t understand what it means to decouple
logic from database code, and I am not sure I understand what Mr.
Bengtsson means by “database code”.

=====


Anders Bengtsson ndrsbngtssn@yahoo.se
Stockholm, Sweden


Gå före i kön och få din sajt värderad på nolltid med Yahoo! Express
Se mer på: http://se.docs.yahoo.com/info/express/help/index.html

hmmm. i think i am guilty of this, and this probably explains my delima. you
seem to know a good deal more about unit testing than i do - it’s always
interesting to read people’s opinions on this as i feel it’s a powerful tool,
but one i don’t completely understand. in any case, it’s very cool the the
framework is now included in ruby. it would be a good thing IMHO if unit
testing becomes a main component of the ‘ruby way’.

-a

···

On Fri, 16 May 2003, [iso-8859-1] Anders Bengtsson wrote:

By database code I meant the parts of the code that knows about the
database. If your application is written in a mostly OO-style you’ll
probably have the “business logic” of the app separated from the gritty
details about how to store a value in a database column. But then again,
most small apps that use a database are built very much around the database,
without that separation. It makes them easier to develop, but less object
oriented, which makes them much harder to unit test. (But other kinds of
testing works ok).

Ara Howard
NOAA Forecast Systems Laboratory
Information and Technology Services
Data Systems Group
R/FST 325 Broadway
Boulder, CO 80305-3328
Email: ara.t.howard@fsl.noaa.gov
Phone: 303-497-7238
Fax: 303-497-7259
====================================

hmmm. i think i am guilty of this, and this probably explains my
delima. you
seem to know a good deal more about unit testing than i do - it’s
always
interesting to read people’s opinions on this as i feel it’s a
powerful tool,
but one i don’t completely understand. in any case, it’s very cool
the the
framework is now included in ruby. it would be a good thing IMHO
if unit
testing becomes a main component of the ‘ruby way’.

I know this is wrought with exceptions, but a very crude rule of
thumb is that if your code is not easily testable, it’s probably
poorly designed, too tightly coupled, not cohesive enough, too big,
etc.; in other words, due for a re-think.

···

Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com

ahoward wrote:

hmmm. i think i am guilty of this, and this probably explains my delima. you
seem to know a good deal more about unit testing than i do - it’s always
interesting to read people’s opinions on this as i feel it’s a powerful tool,
but one i don’t completely understand. in any case, it’s very cool the the
framework is now included in ruby. it would be a good thing IMHO if unit
testing becomes a main component of the ‘ruby way’.

The exciting thing about unit testing is that it’s relatively new, but
at the same time it’s what makes it hard to learn. Lots of good ideas,
but spread out over thousands of list postings and articles.
I would recommend Kent Beck’s “Test Driven Development” book from last
year, I learned a lot from that.

/Anders

···

A n d e r s B e n g t s s o n | ndrsbngtssn@yahoo.se
Stockholm, Sweden |

I would strongly disagree with this. There are some things that
aren’t really able to be broken into clean units for testing. Two of
the projects I’m working on right now are difficult to unit test.
Both of them are involved with creating and manipulating documents
that are intended to be dealt with by other programs. In general,
the documents have to be completely correct, or they aren’t valid at
all.

There are bits that can be tested unit-wise, but they’re smaller
than you might imagine. Neither of these things can be cleanly unit
tested.

I realised that you noted that there are exceptions, but I don’t
think that it’s a safe categorisation at all.

-austin
– Austin Ziegler, austin@halostatue.ca on 2003.05.16 at 13:33:35

···

On Sat, 17 May 2003 00:27:33 +0900, Michael Campbell wrote:

I know this is wrought with exceptions, but a very crude rule of
thumb is that if your code is not easily testable, it’s probably
poorly designed, too tightly coupled, not cohesive enough, too
big, etc.; in other words, due for a re-think.

Austin Ziegler wrote:

There are bits that can be tested unit-wise, but they’re smaller
than you might imagine. Neither of these things can be cleanly unit
tested.

Yes they can, it’s just that no-one has found out how yet. :wink:

I’ve repeatedly been surprised when people have invented ways of unit
testing things that seemed hard or impossible to test.

/Anders

···

A n d e r s B e n g t s s o n | ndrsbngtssn@yahoo.se
Stockholm, Sweden |

Hence my “wrought with exceptions” and “very crude” qualifiers. At
least for what I’m working on, I find that if I can’t test it, it’s
probably worth a re-think. Since I tend to write my tests before I
code (or at least right after), it’s easier to change then. When
you’re working on legacy or pre-existing stuff, its a bit harder.

I find that YOUR characterization of my characterization not being
"safe" interesting; how would you find this dangerous? While it may
not be true in all circumstances, even where it isn’t true, I haven’t
found it “unsafe” in any way. Maybe more time consuming or
suboptimal, but never unsafe.

···

— Austin Ziegler austin@halostatue.ca wrote:

On Sat, 17 May 2003 00:27:33 +0900, Michael Campbell wrote:

I know this is wrought with exceptions, but a very crude rule of
thumb is that if your code is not easily testable, it’s probably
poorly designed, too tightly coupled, not cohesive enough, too
big, etc.; in other words, due for a re-think.

I would strongly disagree with this. There are some things that
aren’t really able to be broken into clean units for testing.


Do you Yahoo!?
The New Yahoo! Search - Faster. Easier. Bingo.
http://search.yahoo.com