This may be off-topic in a Ruby list, although I have noticed that a
lot of folks involved with Ruby are also fond of Unit Testing so it may
not be as off-topic as I think.
Anyway…
Being of the old guard, the ideas behind unit testing are fairly new to
me. I’m giving it a shot just to see how things work out.
I was writing a test today on a class (prototyped in Ruby then
implemented in Objective-C++ for the final application) that manages
the points of a Cubic Bezier curve. The theory of the class was that
the curve is constructed from a series of the classic moveto, lineto,
curveto, and close commands that those folks familiar with the
PostScript drawing model may be familiar with.
The problem is that once you have constructed the path, there is no way
in the public interface (at least not at the moment) for you to go back
and examine the points in the path.
What that means is that in unit testing the drawing commands, there is
no way for a class outside of the BezierContour class to make sure that
the commands were recorded properly short of drawing the path and
comparing the resulting picture with a previously verified picture
(ick!)
One means of making a more meaningful Unit Test would be to allow the
unit test to access the points of the curve (the control polygon for
fellow math geeks) and verify that the resulting polygon matches the
one that the drawing commands should produce. Adding the routine to
examine the control polygon, however, would be a violation of the
encapsulation (or more to the point the current design of the class
which states that the control polygon not be available) of the
BezierContour class.
Now one answer to this conundrum would be to include the methods to
access the control polygon ONLY when the testing code is compiled into
the application. This would be roughly equivalent to having the Ruby
unit testing code extend the BezierContour class with methods to
examine the control polygon.
The question I have, however, is one of style. Is this kind of class
extension often done in Unit Testing or is it more likely that the Unit
Test would only test the common public interface for the class.
Please feel free to respond off this list or cast your responses in a
Ruby Way (with all due respect to Hal) as this is somewhat off-topic.
On Wednesday, September 3, 2003, at 10:42 PM, Scott Thompson wrote:
This may be off-topic in a Ruby list, although I have noticed that a
lot of folks involved with Ruby are also fond of Unit Testing so it
may not be as off-topic as I think.
I know nothing about bezier curves or drawing programs, so this may not
be helpful. I’m also not an expert on unit testing, so others may have
more useful input. Anyway…
[snip]
The problem is that once you have constructed the path, there is no
way in the public interface (at least not at the moment) for you to go
back and examine the points in the path.
[snip]
I think that for unit testing purposes you don’t care whether the
interface is public or private, you test it anyway. I don’t know if
Test::Unit has a mechanism for gaining access to private methods or if
you have to remove privacy for the duration of the tests.
One means of making a more meaningful Unit Test would be to allow the
unit test to access the points of the curve (the control polygon for
fellow math geeks) and verify that the resulting polygon matches the
one that the drawing commands should produce. Adding the routine to
examine the control polygon, however, would be a violation of the
encapsulation (or more to the point the current design of the class
which states that the control polygon not be available) of the
BezierContour class.
I don’t know how your class works, but if it’s monolithic (there is no
interface to the control polygon outside of the object) I think your
choices are (1) to temporarily add an interface for testing purposes or
(2) decouple the elements of the class in order to expose an interface
or (3) write the test to compare the produced object (i.e., by
comparing a previously prepared file with the specified curve to a file
produced by invoking the object with test input data). The last test is
a big leap because it looks at the final product and not intermediate
functionality.
[snip]
The question I have, however, is one of style. Is this kind of class
extension often done in Unit Testing or is it more likely that the
Unit Test would only test the common public interface for the class.
I think the most usual approach is to test all interfaces (public and,
at least initially during development, private) and to decouple
functional elements of objects as much as possible during the
development process. Most important though, is to write a test first
and then implement it and then write another test and so on. If one
writes the test first, concerns about encapsulation usually evaporate
because the only concern in writing the test case is the interface.
[snip]
Regards,
Mark
···
On Wednesday, September 3, 2003, at 10:42 PM, Scott Thompson wrote:
I was writing a test today on a class (prototyped in Ruby then
implemented in Objective-C++ for the final application) that manages
the points of a Cubic Bezier curve. The theory of the class was that
the curve is constructed from a series of the classic moveto, lineto,
curveto, and close commands that those folks familiar with the
PostScript drawing model may be familiar with.
The problem is that once you have constructed the path, there is no way
in the public interface (at least not at the moment) for you to go back
and examine the points in the path.
This might sound silly, but would it be possible to overload, say, #==
so that you can check whether the final state of your object matches
your expectations (with assert_equal)? You’d have to specify a format
to represent the state but at least you’re not exposing the internals
so you would be free to modify them without breaking the tests.
···
On Thu, Sep 04, 2003 at 11:42:04AM +0900, Scott Thompson wrote:
What that means is that in unit testing the drawing commands, there is
no way for a class outside of the BezierContour class to make sure that
the commands were recorded properly short of drawing the path and
========
When I see that I think of mock objects logging all drawing commands.
It might even make sense to just wrap the real objects in a delegator
and log all method calls.
It might sound icky, but isn’t that what unit tests are supposed to
do: verify the output of a class?
If you want to remove the ick, I’d redesign the program to be more
test-friendly. That’s something I’ve faced in the past, and haven’t
really had the opportunity to try out yet. But it’s probably good in
general to prefer test-friendly class design, all else being equal.
Cheers,
Gavin
···
On Thursday, September 4, 2003, 12:42:04 PM, Scott wrote:
What that means is that in unit testing the drawing commands, there is
no way for a class outside of the BezierContour class to make sure that
the commands were recorded properly short of drawing the path and
comparing the resulting picture with a previously verified picture
(ick!)
The problem is that once you have constructed the path, there is no
way in the public interface (at least not at the moment) for you to
go
back and examine the points in the path.
What that means is that in unit testing the drawing commands, there
is
no way for a class outside of the BezierContour class to make sure
that
the commands were recorded properly short of drawing the path and
comparing the resulting picture with a previously verified picture
(ick!)
There are a couple ways I’ve used in the past to solve this sort of
problem.
You don’t test it. I try to avoid this option.
You modify the testee class to allow some sort of querying done
with its innards. Less than ideal. (With Java, however, one can
make these methods “package” visibility, so at least only classes in
the same package can “see” them.)
You make a new method that doesn’t allow inspection of private
data, but allows one to pass IN data, and this method will tell you
if it’s correct. Something like:
class Curve {
private:
Point _points;
…
public:
boolean match(Point testPoints) {
// check here to see if all the testPoints match the values
// in _points
…
};
void otherCoolMethod() { ... };
};
Then your unit test can simply call the method in question, and PASS
IN a set of points that it thinks your curve should have. The Curve
instance then tells you (via the match() method) if that’s right,
rather than having the Curve instance pass its Points out and have
the test class verify them.
I think unit test should test at least the public interface but aren’t
bound to that.
If you have a look at the history of the buzzword unit test, you might
find out that it’s one of those many Smalltalk “inventions” and
Smalltalk doesn’t have private interfaces. By convention you put all
the private methods into a category called “private*” but Smalltalk
doesn’t enforce privacy.
Ok, Ruby isn’t Smalltalk but I strongly believe that in any language
as dynamic as Ruby there are many ways to break encapsulation. You
might consider to use one of them but I recommend that you solve those
problems by choosing one of the following strategies instead:
Use patterns (bridge, strategy, etc…) to reduce the amount
of privat methods. Move the private part of a class into a
public part of a private class.
Use protected instead of private and subclassing to access
those methods.
That’s what I do in C++. (Unit tests aren’t bound to dynamic languages
Don’t use private at all.
That’s what I do in Ruby.
I wouldn’t mind if anybody showed me a more ruby like way
The question I have, however, is one of style. Is this kind of class
extension often done in Unit Testing or is it more likely that the Unit
Test would only test the common public interface for the class.
Most of the time, if you’re having trouble unit-testing a class, that
means the class is doing too much. When I’m coding I find myself
constantly breaking down my code into small manageable classes – not
only does this make my unit-tests short and simple, it also gives me
the added benefit of giving me a clearer, more focused understanding
of what I’m writing.
In the case of your BezierContour class, I have to ask: It generates a
series of PostScript commands (moveto, lineto, etc) and passes them
along. Who receives these commands? Can you write some sort of
MockPostScript that records the commands it receives, and then you can
look into that in your unit test?
In Ruby, this might look like:
class TestBezierCounter << RUNIT::TestCase
def testDraw
counter = BezierCounter.new( < some parameters here > )
mockPostScript = MockPostScript.new
counter.draw( mockPostScript )
assert_equal( MockPostScript::MOVE_TO,
mockPostScript.commands[1].commandType )
end
end
This may be off-topic in a Ruby list, although I have noticed that a
lot of folks involved with Ruby are also fond of Unit Testing so it may
not be as off-topic as I think.
Anyway…
Being of the old guard, the ideas behind unit testing are fairly new to
me. I’m giving it a shot just to see how things work out.
I was writing a test today on a class (prototyped in Ruby then
implemented in Objective-C++ for the final application) that manages
the points of a Cubic Bezier curve. The theory of the class was that
the curve is constructed from a series of the classic moveto, lineto,
curveto, and close commands that those folks familiar with the
PostScript drawing model may be familiar with.
The problem is that once you have constructed the path, there is no way
in the public interface (at least not at the moment) for you to go back
and examine the points in the path.
What that means is that in unit testing the drawing commands, there is
no way for a class outside of the BezierContour class to make sure that
the commands were recorded properly short of drawing the path and
comparing the resulting picture with a previously verified picture
(ick!)
One means of making a more meaningful Unit Test would be to allow the
unit test to access the points of the curve (the control polygon for
fellow math geeks) and verify that the resulting polygon matches the
one that the drawing commands should produce. Adding the routine to
examine the control polygon, however, would be a violation of the
encapsulation (or more to the point the current design of the class
which states that the control polygon not be available) of the
BezierContour class.
Now one answer to this conundrum would be to include the methods to
access the control polygon ONLY when the testing code is compiled into
the application. This would be roughly equivalent to having the Ruby
unit testing code extend the BezierContour class with methods to
examine the control polygon.
The question I have, however, is one of style. Is this kind of class
extension often done in Unit Testing or is it more likely that the Unit
Test would only test the common public interface for the class.
Please feel free to respond off this list or cast your responses in a
Ruby Way (with all due respect to Hal) as this is somewhat off-topic.
The problem is that once you have constructed
the path, there is no way in the public
interface (at least not at the moment) for you
to go back and examine the points in the path.
First off, as other posts have pointed out, these types of problems are
usually avoided by writing the tests first.
Now, to answer your question directly, when a test case needs to dig
into the internals of a class for some reason, I have the test case subclass
the original class, then add the methods that it needs to probe the class.
For example:
Class TestBezierContour < BezierContour
attr_reader :mypath
def getBoundingRect
end
end
Of course you could have the test case just open up the original class
and add these methods, but I find it somehow cleaner to subclass. The main
point is, you can leave the code in the original class pristine, and still
have the test case reopen or subclass it to suit your testing needs.
However, in this case and other cases where a class has a one-way flow
of information to some other non-testable component (operating system,
third-party library, etc.), I think the better way to test is to encapsulate
the non-testable component in its own simple, “thin” class. This also
offers some protection when the non-testable component changes (operating
system upgrade, new release of third-party library). Test cases can now be
written for the original class using a mock object for the non-testable
component. Now, the new “thin” class may still be difficult to test, but in
theory there will not be much code in the class, and simple test cases will
probably suffice.
I heard an interesting take on testing an implementation of an
interface. Your test case is testing the behaviour or state of the implementation, but the rest of the world will only use the
implementation through its interface. So by definition, any client
code will only call interface methods (since they are the only ones it
knows about) so it is irrelevant whether some other,
implementation-specific, methods are public or not.
So your test case can ask all sorts of implementation-specific
questions, and these will be hidden from any client code that only
refers to the object via its interface methods.
It’s another instance of duck-typing - the client only knows it’s a Duck
with a quack method. Only our test case knows it’s a DuckOnAStick and to
test the has_stick? method.
Cheers,
Dan
Michael Campbell wrote:
···
The problem is that once you have constructed the path, there is no
way in the public interface (at least not at the moment) for you to
go
back and examine the points in the path.
What that means is that in unit testing the drawing commands, there
is
no way for a class outside of the BezierContour class to make sure
that
the commands were recorded properly short of drawing the path and
comparing the resulting picture with a previously verified picture
(ick!)
There are a couple ways I’ve used in the past to solve this sort of
problem.
You don’t test it. I try to avoid this option.
You modify the testee class to allow some sort of querying done
with its innards. Less than ideal. (With Java, however, one can
make these methods “package” visibility, so at least only classes in
the same package can “see” them.)
You make a new method that doesn’t allow inspection of private
data, but allows one to pass IN data, and this method will tell you
if it’s correct. Something like:
class Curve {
private:
Point _points;
…
public:
boolean match(Point testPoints) {
// check here to see if all the testPoints match the values
// in _points
…
};
void otherCoolMethod() { … };
};
Then your unit test can simply call the method in question, and PASS
IN a set of points that it thinks your curve should have. The Curve
instance then tells you (via the match() method) if that’s right,
rather than having the Curve instance pass its Points out and have
the test class verify them.
Most of the time, if you’re having trouble unit-testing a class, that
means the class is doing too much.
Well… I don’t think that is the case in this instance. The class
simply records a series of drawing commands (moveto, lineto, curveto,
and close) and at some point in the future can play back that sequence
on a drawing context.
In the case of your BezierContour class, I have to ask: It generates a
series of PostScript commands (moveto, lineto, etc) and passes them
along.
Not exactly. It receives a series of drawing commands and can replay
them into a drawing context later on.
Who receives these commands?
On Operating System drawing context later receives them from the
contour.
Can you write some sort of
MockPostScript that records the commands it receives, and then you can
look into that in your unit test?
Unfortunately, the drawing context that the drawing commands play out
in is owned by the operating system and I don’t have the ability to
duplicate it’s functionality enough to create a mock drawing context
that could verify the points. Perhaps I need to abstract the OS
drawing context one level (at least at testing time) then I could
insert my verification code in the intermediate layer between the
contour class and the operating system’s drawing context.
Most of the time, if you’re having trouble unit-testing a class, that
means the class is doing too much.
Well… I don’t think that is the case in this instance. The class
simply records a series of drawing commands (moveto, lineto, curveto,
and close) and at some point in the future can play back that sequence
on a drawing context.
In the case of your BezierContour class, I have to ask: It generates a
series of PostScript commands (moveto, lineto, etc) and passes them
along.
Not exactly. It receives a series of drawing commands and can replay
them into a drawing context later on.
Who receives these commands?
On Operating System drawing context later receives them from the
contour.
Can you write some sort of
MockPostScript that records the commands it receives, and then you can
look into that in your unit test?
Unfortunately, the drawing context that the drawing commands play out
in is owned by the operating system and I don’t have the ability to
duplicate it’s functionality enough to create a mock drawing context
that could verify the points. Perhaps I need to abstract the OS
drawing context one level (at least at testing time) then I could
insert my verification code in the intermediate layer between the
contour class and the operating system’s drawing context.
This isn’t directly related to this particular problem, but here is an
open question for the group:
In this particular case, there are some good ideas of how to unit test it.
But in general, it seems like things start breaking down the closer you
get to a one-way data transfer. In this case, it’s tricky because we say,
well, once we pass it to the operating system’s drawing context, how do we
know it’s correct afterward? We can test it before hand with many middle
layers, but we can’t [easily] test it afterward.
What about some other extreme cases? Like, a class that plays a sound on a
speaker? How do you unit test that? Or, likewise, any class that
interfaces with the “world” “outside” your control? What if they are so
complex that you can’t easily or acurately model them? What about output
that “needs” human interaction?
Anyway, I know what my answers to these questions have been on some
projects (and unfortunately enough, it’s often “just don’t bother
testing”), but I’m curious what people’s thoughts on this are. =)
Can you write some sort of
MockPostScript that records the commands it receives, and then you can
look into that in your unit test?
Unfortunately, the drawing context that the drawing commands play out
in is owned by the operating system and I don’t have the ability to
duplicate it’s functionality enough to create a mock drawing context
that could verify the points. Perhaps I need to abstract the OS
drawing context one level (at least at testing time) then I could
insert my verification code in the intermediate layer between the
contour class and the operating system’s drawing context.
I’d recommend doing this exactly. After a while, any system I’ve
worked on for a while ends up having abstractions for almost every
sort of external service – I do a lot of database-backed web sites so
in my case this tends to be things like the filesystem, SMTP, CGI
field input, etc. Then you can write mock objects that mimic the
interface but just record the commands they receive so your unit tests
can ask what happened. This is more work up front, of course. But I
find this process essential to getting that feeling of confidence
testing gives you.
What about some other extreme cases? Like, a class that plays a sound on a
speaker? How do you unit test that? Or, likewise, any class that
interfaces with the “world” “outside” your control? What if they are so
complex that you can’t easily or acurately model them? What about output
that “needs” human interaction?
I also regularly deal with cases that involve interfacing to external
systems.
If I want to test something that grabs a web page, what happens if due
to network issues, my access times out? If what you’re really
interested in is not related to the network, then you can either test to
see if you have the right URL and headers to fetch the web page, or fake
the fetch and see if your code handles the data properly, but what if
the web access is essential?
An example would be a bit of code that tries to use some data to figure
out what URL to use to get what it needs. The only way to test to see
if that URL is correct is to see if the web page it fetches contains the
data it expects. Because of the nature of the problem (the changing
web), you can’t tell by looking at the URL if it is correct, you need to
see what it fetches. You could always retry if a network error occurs,
but what if the error is a 500 series HTTP error (i.e. your URL was
correct but the server is temporarily broken)?
Are there general unit testing strategies for dealing with things
outside the system you have control over?
Are there general unit testing strategies for dealing with things
outside the system you have control over?
You can always test the failure cases, too – and when you find one you
haven’t tested, add it to the suite.
Fake a 500. Fake a 404. Fake a timeout – throw an exception that
could happen in your code. Send it bogus data. Send it good data. All
should pass tests. Use your intuition about what will possibly break,
and if you can, think of /all/ the ways it can break, if it’s a
documented enough system.
What about some other extreme cases? Like, a class that plays a sound on
a
speaker? How do you unit test that? Or, likewise, any class that
interfaces with the “world” “outside” your control? What if they are so
complex that you can’t easily or acurately model them? What about output
that “needs” human interaction?
I also regularly deal with cases that involve interfacing to external
systems.
If I want to test something that grabs a web page, what happens if due
to network issues, my access times out? If what you’re really
interested in is not related to the network, then you can either test to
see if you have the right URL and headers to fetch the web page, or fake
the fetch and see if your code handles the data properly, but what if
the web access is essential?
Just to add some more fuel to the discussion: another real general place
where this comes up a lot is graphics applications.
From a real raytracer project I did: How do I know that my raytracer
rendered a sphere correctly? Well, I look at the output image and say
“hmmm… yeah, it looks like a 3d sphere…” – there were a lot of things
like that. I could unit test the guts, but often writing the tests
themselves required first writing a little program to compute what the
results should be…
Suffice to say, on that particular project, I ended up just unit testing
some core pieces (where, arguably it was simple enough that I didn’t
really need the tests) and just doing ad-hoc testing on the larger pieces
that did complex math or actual rendering (but, ironically, this is where
I would have benefitted the most from test-first unit-tests, because it
was so hard to get right!)
An example would be a bit of code that tries to use some data to figure
out what URL to use to get what it needs. The only way to test to see
if that URL is correct is to see if the web page it fetches contains the
data it expects. Because of the nature of the problem (the changing
web), you can’t tell by looking at the URL if it is correct, you need to
see what it fetches. You could always retry if a network error occurs,
but what if the error is a 500 series HTTP error (i.e. your URL was
correct but the server is temporarily broken)?
Are there general unit testing strategies for dealing with things
outside the system you have control over?
I think it’s pretty clear that there is set of “un-unit-testable” things.
I wonder, though, if anyone has some brilliant ideas on how to mitigate
this class of problems. So far what I have in my head is:
Don’t bother unit testing (yikes!).
Do regression tests with “golden” results (i.e. compare against
previous output – this assumes no test-first, and that you at some point
got it right and verified by hand–this, IMO, is a lot to assume)
Make use of mock-objects to simulate an external interface (only works
if it’s low-enough complexity for you to model)
What about some other extreme cases? Like, a class that plays a sound on a
speaker? How do you unit test that? Or, likewise, any class that
interfaces with the “world” “outside” your control? What if they are so
complex that you can’t easily or acurately model them? What about output
that “needs” human interaction?
I’ve asked similar questions when unit-testing screen-scraping applications.
In this case, I’m sending a set of commands to an external system, but
there’s no way I can prove with a unit test that the commands I send have
the effect that I want.
I tried to tackle this by examining the output of my class using a mock
object that would just check that the commands sent to it were the ones
it expected. Then at least one test would have to be done by hand,
ensuring that sending those commands to the external system had the desired
effect.
One limitation of this way of doing things is that the class could be
changed in the future to operate in a way that is semantically equivalent
from the point of view of the external system (e.g. sending commands
in a different order), but this would still cause the test to break.
Arguably, this is no bad thing - we’re now getting false positives reported
but this does indicate that we need to explicitly check the test by hand
again to ensure that what we have done is correct. On the whole I think
this is better than having no tests.
Fake a 500. Fake a 404. Fake a timeout – throw an exception that
could happen in your code. Send it bogus data. Send it good data. All
should pass tests.
Sure that’s easy. But what happens if you can’t test the “success” case
reliably because it depends on things outside your control?
In my example, I can:
Test to see if I can generate a URL
Test to see if the URL is syntactically valid
Test to see if I can handle all the various ways in which accessing
that URL can fail
Test to see if I can handle all the various types of data I expect
the URL to return if it does respond
But what if I want to test to see if the URL I generated points to a
valid web page? The only way I know of to test that is to go to that
URL and see if there’s a web page there. The problem is that you can’t
prove that by counterexample. Just because I get an error while trying
to access the page doesn’t mean the URL is invalid, it could just be
another type of error.
If I get a DNS lookup error, it could be that the hostname in the URL is
wrong, or it could just be a hiccup in the DNS server. If I get a 404
error, it could be that a transparent proxy along the way broke, or that
the server I’m contacting was temporarily misconfigured.
Is there a good way of testing “does the URL I generated point to a
valid web page”? For the more general case, is there a good way of
testing when a failure doesn’t necessarily mean that your code is wrong?
