Hi all! I am afraid this post is a bit offtopic. If you are not interested in Bugtracking and Unit Tests, read no further! 
At the moment I am playing with the idea of writing a simple bugtracking system in Ruby. I thought that it would be a cool idea to integrate Unit Tests with such a system: Every reported bug should get a unit test assigned. If a unit test fails, the bug is automatically reopened. When the tests is successful, it is closed. I have not really thought this through, I am just asking if people here think this is be a good idea or not.
martinus
Actually, this does sound like a great idea. Idea: allow the users who submit
bugs to provide the unit test to prove the bug's existence.
Sean O'Dell
路路路
On Tuesday 06 July 2004 11:17, martinankerl at eml dot cc wrote:
Hi all! I am afraid this post is a bit offtopic. If you are not
interested in Bugtracking and Unit Tests, read no further! 
At the moment I am playing with the idea of writing a simple bugtracking
system in Ruby. I thought that it would be a cool idea to integrate Unit
Tests with such a system: Every reported bug should get a unit test
assigned. If a unit test fails, the bug is automatically reopened. When
the tests is successful, it is closed. I have not really thought this
through, I am just asking if people here think this is be a good idea or
not.
martinankerl at eml dot cc wrote:
Hi all! I am afraid this post is a bit offtopic. If you are not interested in Bugtracking and Unit Tests, read no further!
At the moment I am playing with the idea of writing a simple bugtracking system in Ruby. I thought that it would be a cool idea to integrate Unit Tests with such a system: Every reported bug should get a unit test assigned. If a unit test fails, the bug is automatically reopened. When the tests is successful, it is closed. I have not really thought this through, I am just asking if people here think this is be a good idea or not.
martinus
The best bug/issue tracking tool I've found is Roundup which is written in Python:
http://roundup.sf.net
It is incredibly easy to customize both the data structure, page layout and reports. Currently using it for todos, issue tracking and project management (3 different configurations).
If you can do something like this in Ruby, I'm sure it'll be a hit!
Seems like a great way to prevent bugs from coming back to life. The Pragmatic Programmer talks about the importance of adding tests for each bug that is discovered. If you had a test script for every bug that popped up you would end up with lots of files. Although I guess a test script could just be a string/record in a database which is used to generate a temp file when the tests actually needs to be run (or just pipes a script into ruby). Using a database would also help solve the problem of creating a link between the bug report and the unit test....
Sounds like a major project. I would be interested in the results.
-Charlie
路路路
On Jul 6, 2004, at 11:17 AM, martinankerl at eml dot cc wrote:
Hi all! I am afraid this post is a bit offtopic. If you are not interested in Bugtracking and Unit Tests, read no further!
At the moment I am playing with the idea of writing a simple bugtracking system in Ruby. I thought that it would be a cool idea to integrate Unit Tests with such a system: Every reported bug should get a unit test assigned. If a unit test fails, the bug is automatically reopened. When the tests is successful, it is closed. I have not really thought this through, I am just asking if people here think this is be a good idea or not.
martinus
What if the test fails for a different reason that that which triggered the original bug report? Shouldn't that be a new bug report?
I'm not quite sure what the benefit of automatically reopening a bug would be anyway. If you don't check in code unless it passes all unit tests, then you'd never have to worry about this situation in the first place.
I generally think that it is more worthwhile to expend effort on tests than on a bug tracking system.
路路路
At 01:17 PM 7/6/2004, martinankerl at eml dot cc wrote:
If a unit test fails, the bug is automatically reopened.
______________________________________
Bret Pettichord, Software Tester
Consultant - www.pettichord.com
Author - www.testinglessons.com
Blogger - www.io.com/~wazmo/blog
Scripting Web Tests Tutorial
XP Agile Universe, Aug 15, Calgary
http://www.xpuniverse.com/schedule/T15
Hello Randy,
martinankerl at eml dot cc wrote:
Hi all! I am afraid this post is a bit offtopic. If you are not
interested in Bugtracking and Unit Tests, read no further!
At the moment I am playing with the idea of writing a simple bugtracking
system in Ruby. I thought that it would be a cool idea to integrate Unit
Tests with such a system: Every reported bug should get a unit test
assigned. If a unit test fails, the bug is automatically reopened. When
the tests is successful, it is closed. I have not really thought this
through, I am just asking if people here think this is be a good idea or
not.
martinus
The best bug/issue tracking tool I've found is Roundup which is written
in Python:
http://roundup.sf.net
I found that a AskSam is the best program for tracking
bugs and features. And for 99% of all projects that are done with
scripting languages a "bugs.txt" is an appropriate bugtracking tool.
And to be honest i stopped unit tests, because the current libraries
GUI, networking, filesystems do not support them well enough. Hope
that this will improve in the future.
路路路
--
Best regards, emailto: scholz at scriptolutions dot com
Lothar Scholz http://www.ruby-ide.com
CTO Scriptolutions Ruby, PHP, Python IDE 's
What if the test fails for a different reason that that which triggered the
original bug report? Shouldn't that be a new bug report?
I finally had a bit time to look a bit further into this project. The
problem you state is, that the unit test might not directly related to
the report: it can cover more or less of the problem. There actually
seems to be an M to N relation between bugs and unit tests. This
complicates things, but I find this very interesting: now there is
information available that enables me to list all the tests that
relate to a bug, and the other way round. I am not yet shure what this
information is good for, but I will think of something
I'm not quite sure what the benefit of automatically reopening a bug would
be anyway. If you don't check in code unless it passes all unit tests, then
you'd never have to worry about this situation in the first place.
That's true. But I am not shure that this is always possible. There
might be bug reports that take a lot of time to fix. Here's again the
problem that unit tests do not directly relate to bug reports. Hm, I
need more time for this
I generally think that it is more worthwhile to expend effort on tests than
on a bug tracking system.
Well, I want to spend effort on the combination of both. My goal is to
make both more convenient and useful, and maybe create something
that's interesting, and new.
Martin
On the other hand, it would save a lot of time trying to decipher cryptic bug
reports, or reports that misinterpret the nature of the bug itself, making
actual location of the problem difficult. I would say at least half, if not
more, of all bug reports I get don't come with enough information for me to
locate the problem, or I simply cannot reproduce the bug as described.
Sean O'Dell
路路路
On Tuesday 06 July 2004 11:31, Jamis Buck wrote:
Sean O'Dell wrote:
> On Tuesday 06 July 2004 11:17, martinankerl at eml dot cc wrote:
>>Hi all! I am afraid this post is a bit offtopic. If you are not
>>interested in Bugtracking and Unit Tests, read no further!
>>
>>At the moment I am playing with the idea of writing a simple bugtracking
>>system in Ruby. I thought that it would be a cool idea to integrate Unit
>>Tests with such a system: Every reported bug should get a unit test
>>assigned. If a unit test fails, the bug is automatically reopened. When
>>the tests is successful, it is closed. I have not really thought this
>>through, I am just asking if people here think this is be a good idea or
>>not.
>
> Actually, this does sound like a great idea. Idea: allow the users who
> submit bugs to provide the unit test to prove the bug's existence.
It would have to be handled delicately. Otherwise, this sounds like a
great way to discourage people from submitting bugs. 
I can think of
lots of people, many of whom I work with, who would rather just work
around the bug than report it if the bug reporting process was too
cumbersome.
Most things aren't that bad. Mocking up files, mocking up command line interactions, these are cheap. GUI's are tough to test. I think there is alot of hope for web libraries though. On the protocol level, HTTP behaves alot like a command-line. It's a laborious, rather than impossible, task to make HTTP easy to test.
Cheers,
Patrick
路路路
On Tuesday, July 6, 2004, at 04:55 PM, Lothar Scholz wrote:
And to be honest i stopped unit tests, because the current libraries
GUI, networking, filesystems do not support them well enough. Hope
that this will improve in the future.
Sean O'Dell wrote:
It would have to be handled delicately. Otherwise, this sounds like a
great way to discourage people from submitting bugs. I can think of
lots of people, many of whom I work with, who would rather just work
around the bug than report it if the bug reporting process was too
cumbersome.
On the other hand, it would save a lot of time trying to decipher cryptic bug reports, or reports that misinterpret the nature of the bug itself, making actual location of the problem difficult. I would say at least half, if not more, of all bug reports I get don't come with enough information for me to locate the problem, or I simply cannot reproduce the bug as described.
Sean O'Dell
To be sure, there is a trade off. In this case, it's a trade off between having really good reports for the bugs that get reported, but missing many real bugs that would have been reported if the process were less cumbersome, and having poor-to-mediocre bug reports that, when reproducible, cover a larger span of bugs.
I know exactly what you mean, Sean--I've recieved the same quality of bug reports for many projects, and it can be frustrating. But I'd prefer to know that people have having problems, than to say, "gee, no one is reporting bugs, therefore my software must be working as expected," when that is not really the case.
路路路
--
Jamis Buck
jgb3@email.byu.edu
http://www.jamisbuck.org/jamis
"I use octal until I get to 8, and then I switch to decimal."
You've cut out my quote where I said "allow the user"; I didn't say or imply
"force the user." There are many users who can report bugs but simply don't
have the programming skill required to submit a unit test. That said, to a
person with the skill to provide a unit test, it would be a simple process to
write "x feature bombs, check this out" followed by a short unit test.
Sean O'Dell
路路路
On Tuesday 06 July 2004 11:58, Jamis Buck wrote:
Sean O'Dell wrote:
>>It would have to be handled delicately. Otherwise, this sounds like a
>>great way to discourage people from submitting bugs. I can think of
>>lots of people, many of whom I work with, who would rather just work
>>around the bug than report it if the bug reporting process was too
>>cumbersome.
>
> On the other hand, it would save a lot of time trying to decipher cryptic
> bug reports, or reports that misinterpret the nature of the bug itself,
> making actual location of the problem difficult. I would say at least
> half, if not more, of all bug reports I get don't come with enough
> information for me to locate the problem, or I simply cannot reproduce
> the bug as described.
To be sure, there is a trade off. In this case, it's a trade off between
having really good reports for the bugs that get reported, but missing
many real bugs that would have been reported if the process were less
cumbersome, and having poor-to-mediocre bug reports that, when
reproducible, cover a larger span of bugs.
The thing I like about the idea of having actual code which is being
used to test bugs in an issue tracker is that you have a "live" suite
of tests which could catch the re-appearance of bugs even before a
user experiences them.
How about this: each new bug reporter has a status method
automatically associated with it, which is a test case for that issue.
The default status code always returns an error, so before an issue
can be closed, someone must either manually write a test case which
can then be checked to determine the issue status, or they must
explicitly reassign a different "stock" status method which does not
raise an exception (and possibly returns some symbolic value like
"WorkingAsDesigned", or "NotReproduceable", which can go into a full
test report).
Unforunately, it seems like it would require a huge amount of trust in
your users, or tightly-restricted permissions for who could write such
cases, since you would be (presumably) accepting live code through the
web, with all of the security implications that causes.
Lennon
How about this: each new bug reporter has a status method
automatically associated with it, which is a test case for that issue.
The default status code always returns an error, so before an issue
can be closed, someone must either manually write a test case which
can then be checked to determine the issue status, or they must
explicitly reassign a different "stock" status method which does not
raise an exception (and possibly returns some symbolic value like
"WorkingAsDesigned", or "NotReproduceable", which can go into a full
test report).
This is an excellent idea, even if combined with a system that allows users to
submit their own unit tests. The user-submitted unit test could be disable
until a developer has looked at it for security and then it could be added to
the battery of tests.
Unforunately, it seems like it would require a huge amount of trust in
your users, or tightly-restricted permissions for who could write such
cases, since you would be (presumably) accepting live code through the
web, with all of the security implications that causes.
I think perhaps the cadre of developers for a given project could all be given
the ability to review and "enable" user-submitted unit tests.
Sean O'Dell
路路路
On Tuesday 06 July 2004 12:31, Lennon Day-Reynolds wrote: