# Test::Unit run order **URL:** \ **Category:** ruby-talk\ **Created:** [15 July 2009 17:15 UTC](https://rubytalk.org/t/test-unit-run-order/54423 "2009-07-15T17:15:14Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Dylan](https://avatars.discourse-cdn.com/v4/letter/d/e95f7d/32.png) [@Dylan](https://rubytalk.org/u/Dylan)\ **Post date:** [15 July 2009 17:15 UTC](https://rubytalk.org/t/test-unit-run-order/54423/1 "2009-07-15T17:15:14Z")
Hey there! I had a question about the order tests are run in Test::Unit. I have a class with about 10 tests and I'd like to be able to better control the order they run in. Right now they just run alphabetically, and I could just add prefixes to my test names, but I'm likely not going to be the one updating this code in the future and I'd like to keep it as straightforward and simple as possible (i.e. not end up with test\_admintests\_part1\_login as a test name). I'm keeping them all in one class because I'm using CI\_reporter to generate the XML results and I'd like it to be in one file. Is there any way to change the order individual tests run, or is there a way to have CI\_reporter create only 1 XML file for multiple test suites. I've looked around quite a bit for answers to either of these questions but have so far found nothing (although it's a distinct possibility that I just totally missed it an need a smack on the head). 馃檪 Thank you very much! -Dylan ---
**Author:** ![James\_Coglan2](https://avatars.discourse-cdn.com/v4/letter/j/c67d28/32.png) [@James\_Coglan2](https://rubytalk.org/u/James_Coglan2)\ **Post date:** [15 July 2009 17:33 UTC](https://rubytalk.org/t/test-unit-run-order/54423/2 "2009-07-15T17:33:27Z")
I can't speak about your XML stuff, but I'm pretty sure the Test::Unit::TestCase.suite method sorts the tests by name so you can't change the order. Your options are: 1) Construct the test suite yourself rather than relying on the above method to do it for you. You'll have to dig around in Test::Unit's source to figure out how to run tests without using AutoRunner. 2) Take a look at some of the code in test/unit/ui and look at the logging events that tests fire. These should allow you to collect event data and reorder it however you like before creating a report. Both will require some inspecting of Test::Unit's source, but I'm sure others have better suggestions. Test::Unit's README is pretty good if you need to know some details on its internals, e.g. how TestCases are converted to suites and run. > **路路路** > > 2009/7/15 Dylan \ > > > Hey there! I had a question about the order tests are run in > > Test::Unit. I have a class with about 10 tests and I'd like to be able > > to better control the order they run in. Right now they just run > > alphabetically, and I could just add prefixes to my test names, but > > I'm likely not going to be the one updating this code in the future > > and I'd like to keep it as straightforward and simple as possible > > (i.e. not end up with test\_admintests\_part1\_login as a test name). I'm > > keeping them all in one class because I'm using CI\_reporter to > > generate the XML results and I'd like it to be in one file. Is there > > any way to change the order individual tests run, or is there a way to > > have CI\_reporter create only 1 XML file for multiple test suites. I've > > looked around quite a bit for answers to either of these questions but > > have so far found nothing (although it's a distinct possibility that I > > just totally missed it an need a smack on the head). 馃檪 Thank you very > > much! ---
**Author:** ![Daniel\_Berger2](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/daniel_berger2/32/2243_2.png) [@Daniel\_Berger2](https://rubytalk.org/u/Daniel_Berger2)\ **Post date:** [16 July 2009 13:41 UTC](https://rubytalk.org/t/test-unit-run-order/54423/3 "2009-07-16T13:41:20Z")
I submitted a feature request for Test::Unix 2.x for something along these lines and it was accepted. [http://rubyforge.org/tracker/index.php?func=detail&aid=26627&group\_id=5650&a](http://rubyforge.org/tracker/index.php?func=detail&aid=26627&group_id=5650&a) tid=21859 In short, Kouhei added the '--order' option which lets you run tests in the order in which the methods are defined within the test suite rather than alphabetically. It hasn't been released yet, but you can grab it from CVS (or wait until it is released, though I'm not sure when that will be). Regards, Dan > **路路路** > > > -----Original Message----- > > From: Dylan [mailto:mejari@gmail.com] > > Sent: Wednesday, July 15, 2009 11:15 AM > > To: ruby-talk ML > > Subject: Test::Unit run order > > > > Hey there! I had a question about the order tests are run in > > Test::Unit. I have a class with about 10 tests and I'd like to be able > > to better control the order they run in. Right now they just run > > alphabetically, and I could just add prefixes to my test names, but > > I'm likely not going to be the one updating this code in the future > > and I'd like to keep it as straightforward and simple as possible > > (i.e. not end up with test\_admintests\_part1\_login as a test name). I'm > > keeping them all in one class because I'm using CI\_reporter to > > generate the XML results and I'd like it to be in one file. Is there > > any way to change the order individual tests run, or is there a way to > > have CI\_reporter create only 1 XML file for multiple test suites. I've > > looked around quite a bit for answers to either of these questions but > > have so far found nothing (although it's a distinct possibility that I > > just totally missed it an need a smack on the head). 馃檪 Thank you very > > much! ---
**Author:** ![Ryan\_Davis1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/ryan_davis1/32/1848_2.png) [@Ryan\_Davis1](https://rubytalk.org/u/Ryan_Davis1)\ **Post date:** [16 July 2009 19:16 UTC](https://rubytalk.org/t/test-unit-run-order/54423/4 "2009-07-16T19:16:41Z")
order dependent tests are a bug. every single test should be able to run and pass in isolation. eg: % ruby -Ilib test/test\_something.rb -n test\_one\_test should always pass no matter what the file or test name is. > **路路路** > > On Jul 15, 2009, at 10:15 , Dylan wrote: > > > Hey there! I had a question about the order tests are run in > > Test::Unit. I have a class with about 10 tests and I'd like to be able > > to better control the order they run in. Right now they just run > > alphabetically [...] ---
**Author:** ![Dylan](https://avatars.discourse-cdn.com/v4/letter/d/e95f7d/32.png) [@Dylan](https://rubytalk.org/u/Dylan)\ **Post date:** [16 July 2009 22:30 UTC](https://rubytalk.org/t/test-unit-run-order/54423/5 "2009-07-16T22:30:15Z")
Which is nice in theory, but not always the best option. In my case, I'm testing a web app, and I need to test different things in different logins, so I'd like to order it so I log in to a normal user account, run some tests, then log in to an admin account, run some other tests, etc... without having to re-login for every test, which wastes a lot of time and doesn't give me anything in return other than being able to say "Ooh, my tests are independent" (I already have a separate test for the login system). If the login system doesn't work, none of the tests are going to pass even if I had separate login scripts for each one. So I either need a way to order the tests so I can use certain logins for certain tests, or I need a way to create only 1 XML file from CI\_reporter from multiple suites, then I can just set up 1 suite for each set of tests (this would be preferred, although a way to order the tests would still be helpful, at least as a placeholder to use during test development). @Daniel: Thanks I will check that out. -Dylan > **路路路** > > On Jul 16, 12:16 pm, Ryan Davis \ wrote: > > > On Jul 15, 2009, at 10:15 , Dylan wrote: > > > > \> Hey there! I had a question about the order tests are run in > > \> Test::Unit. I have a class with about 10 tests and I'd like to be able > > \> to better control the order they run in. Right now they just run > > \> alphabetically [...] > > > > order dependent tests are a bug. every single test should be able to > > run and pass in isolation. eg: > > > > % ruby -Ilib test/test\_something.rb -n test\_one\_test > > > > should always pass no matter what the file or test name is. ---
**Author:** ![Kouhei\_Sutou1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/kouhei_sutou1/32/2034_2.png) [@Kouhei\_Sutou1](https://rubytalk.org/u/Kouhei_Sutou1)\ **Post date:** [16 July 2009 23:56 UTC](https://rubytalk.org/t/test-unit-run-order/54423/6 "2009-07-16T23:56:57Z")
Hi, > **路路路** > > 2009/7/16 Daniel Berger \: > > > I submitted a feature request for Test::Unix 2.x for something along these > > lines and it was accepted. > > > > [http://rubyforge.org/tracker/index.php?func=detail&aid=26627&group\_id=5650&a](http://rubyforge.org/tracker/index.php?func=detail&aid=26627&group_id=5650&a) > > tid=21859 > > > > In short, Kouhei added the '--order' option which lets you run tests in the > > order in which the methods are defined within the test suite rather than > > alphabetically. > > > > It hasn't been released yet, but you can grab it from CVS (or wait until it > > is released, though I'm not sure when that will be). > > I'll release a new version of test-unit in RubyKaigi2009. 馃檪 > (I'll release it on today or tomorrow.) ---
**Author:** ![Kouhei\_Sutou1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/kouhei_sutou1/32/2034_2.png) [@Kouhei\_Sutou1](https://rubytalk.org/u/Kouhei_Sutou1)\ **Post date:** [17 July 2009 00:01 UTC](https://rubytalk.org/t/test-unit-run-order/54423/7 "2009-07-17T00:01:47Z")
Hi, > **路路路** > > 2009/7/17 Ryan Davis \: > > > order dependent tests are a bug. every single test should be able to run and > > pass in isolation. eg: > > > > % ruby -Ilib test/test\_something.rb -n test\_one\_test > > > > should always pass no matter what the file or test name is. > > I think so too. So I've also added "--order=random" option. > > Thanks, > -- > kou ---
**Author:** ![Ryan\_Davis1](https://yyz1.discourse-cdn.com/flex029/user_avatar/rubytalk.org/ryan_davis1/32/1848_2.png) [@Ryan\_Davis1](https://rubytalk.org/u/Ryan_Davis1)\ **Post date:** [17 July 2009 04:55 UTC](https://rubytalk.org/t/test-unit-run-order/54423/8 "2009-07-17T04:55:06Z")
...or you could deal with the real problem of having slow tests and not the symptoms of it. I've talked about this a lot in the past[\*], so I'm not going to repeat myself here. I'll simply reassert:    \_all\_ tests should \_always\_ pass when run by themselves. and I'll expand on that:    \_all\_ tests should run fast enough that they're a pleasure to run \_all\_ the time. [\*] See my talks on Ruby Sadism, Electric Kool-ade Acid Testing, and many others for related infos. > **路路路** > > On Jul 16, 2009, at 15:30 , Dylan wrote: > > > On Jul 16, 12:16 pm, Ryan Davis \ wrote: > > > > > On Jul 15, 2009, at 10:15 , Dylan wrote: > > > > > > > Hey there! I had a question about the order tests are run in > > > > Test::Unit. I have a class with about 10 tests and I'd like to be able > > > > to better control the order they run in. Right now they just run > > > > alphabetically [...] > > > > > > order dependent tests are a bug. every single test should be able to > > > run and pass in isolation. eg: > > > > > > % ruby -Ilib test/test\_something.rb -n test\_one\_test > > > > > > should always pass no matter what the file or test name is. > > > > Which is nice in theory, but not always the best option. In my case, > > I'm testing a web app, and I need to test different things in > > different logins, so I'd like to order it so I log in to a normal user > > account, run some tests, then log in to an admin account, run some > > other tests, etc... without having to re-login for every test, which > > wastes a lot of time and doesn't give me anything in return other than > > being able to say "Ooh, my tests are independent" (I already have a > > separate test for the login system). If the login system doesn't work, > > none of the tests are going to pass even if I had separate login > > scripts for each one. > > > > So I either need a way to order the tests so I can use certain logins > > for certain tests, or I need a way to create only 1 XML file from > > CI\_reporter from multiple suites [...] ---
**Author:** ![Dylan](https://avatars.discourse-cdn.com/v4/letter/d/e95f7d/32.png) [@Dylan](https://rubytalk.org/u/Dylan)\ **Post date:** [20 July 2009 16:50 UTC](https://rubytalk.org/t/test-unit-run-order/54423/9 "2009-07-20T16:50:13Z")
I would love to do that, but I have no idea how. The "real problem" of my slow tests is caused by the very real requirement of logging in every time, were they to be truly independent. I'm not in charge of the bandwidth to the server, I'm not in charge of the log-in code, and I'm not in charge of the "Lets make the tests take less than a day to run" requirement. I'm testing a web app, and web apps by their very nature have to transmit data back and forth, which takes more time than any local machine doing operations. In fact, the "slow" tests aren't all that slow were it a real user in a real situation. Running so many tests takes long enough as it is, and I still don't see the benefit of independence in my case (even so, I would still prefer it, as I've said, but in my situation the tradeoffs do not seem to be worth it. Nothing further gets tested, they run just the same as if I logged in every time unless there is some major screw up with cookies/ cache, and the other tests aren't going to screw up because of a previous one (or on the off-chance they do it would be easily traceable back to the actual problem). Yes, I do realize that there is a chance that not re-logging in every time might screw up some test, but the chance of it is remote enough, and the problems that would occur if it did happen are small enough, that the added benefit of being able to get results back relatively quickly is (to me) worth it. I am in fact trying to solve your second rule by removing the first. Were I to make them truly independent it would take well over 4-5 times as long to run, which means no one would bother running them/ waiting around for their results, which makes them useless. I would welcome a solution that could fulfill both your rules there, but until I find one I have to err on the side of usability. -Dylan > **路路路** > > On Jul 16, 9:55 pm, Ryan Davis \ wrote: > > > On Jul 16, 2009, at 15:30 , Dylan wrote: > > > > \> On Jul 16, 12:16 pm, Ryan Davis \ wrote: > > \>\> On Jul 15, 2009, at 10:15 , Dylan wrote: > > > > \>\>\> Hey there! I had a question about the order tests are run in > > \>\>\> Test::Unit. I have a class with about 10 tests and I'd like to be > > \>\>\> able > > \>\>\> to better control the order they run in. Right now they just run > > \>\>\> alphabetically [...] > > > > \>\> order dependent tests are a bug. every single test should be able to > > \>\> run and pass in isolation. eg: > > > > \>\> % ruby -Ilib test/test\_something.rb -n test\_one\_test > > > > \>\> should always pass no matter what the file or test name is. > > > > \> Which is nice in theory, but not always the best option. In my case, > > \> I'm testing a web app, and I need to test different things in > > \> different logins, so I'd like to order it so I log in to a normal user > > \> account, run some tests, then log in to an admin account, run some > > \> other tests, etc... without having to re-login for every test, which > > \> wastes a lot of time and doesn't give me anything in return other than > > \> being able to say "Ooh, my tests are independent" (I already have a > > \> separate test for the login system). If the login system doesn't work, > > \> none of the tests are going to pass even if I had separate login > > \> scripts for each one. > > > > \> So I either need a way to order the tests so I can use certain logins > > \> for certain tests, or I need a way to create only 1 XML file from > > \> CI\_reporter from multiple suites [...] > > > > ..or you could deal with the real problem of having slow tests and > > not the symptoms of it. I've talked about this a lot in the past[\*], > > so I'm not going to repeat myself here. I'll simply reassert: > > > > \_all\_ tests should \_always\_ pass when run by themselves. > > > > and I'll expand on that: > > > > \_all\_ tests should run fast enough that they're a pleasure to run > > \_all\_ the time. > > > > [\*] See my talks on Ruby Sadism, Electric Kool-ade Acid Testing, and > > many others for related infos. ---
**Author:** ![Rick\_DeNatale1](https://avatars.discourse-cdn.com/v4/letter/r/bbce88/32.png) [@Rick\_DeNatale1](https://rubytalk.org/u/Rick_DeNatale1)\ **Post date:** [20 July 2009 17:36 UTC](https://rubytalk.org/t/test-unit-run-order/54423/10 "2009-07-20T17:36:11Z")
It sounds to me that there's a confusion here between unit-testing and full-stack testing. For example if this were a rails app, the normal approach is to have tests for models, separate tests for controllers, and integration tests to test user interface flows. And even in the last case, this is normally done not by transmitting data back and forth, instead Rails provides special integration test mocks for the http infrastructure which simulates actual http request and response transmission. And logging in (i.e. authorization) is one thing which is usually mocked out for many if not all tests. Jim Weirich does a humorous 'skit' about mocking login, in which he has to help a co-worker who was testing failed password validation using real access and his own account, and kept getting locked out of the system. Less traditional ruby web developers are using tools like Cucumber and webrat to do integration/acceptance testing, but these are still simulating infrastructure in different ways. > **路路路** > > On Mon, Jul 20, 2009 at 12:50 PM, Dylan\ wrote: > > > I would love to do that, but I have no idea how. The "real problem" of > > my slow tests is caused by the very real requirement of logging in > > every time, were they to be truly independent. I'm not in charge of > > the bandwidth to the server, I'm not in charge of the log-in code, and > > I'm not in charge of the "Lets make the tests take less than a day to > > run" requirement. I'm testing a web app, and web apps by their very > > nature have to transmit data back and forth, which takes more time > > than any local machine doing operations. > > -- > Rick DeNatale > > Blog: [http://talklikeaduck.denhaven2.com/](http://talklikeaduck.denhaven2.com/) > Twitter: [http://twitter.com/RickDeNatale](http://twitter.com/RickDeNatale) > WWR: [http://www.workingwithrails.com/person/9021-rick-denatale](http://www.workingwithrails.com/person/9021-rick-denatale) > LinkedIn: [http://www.linkedin.com/in/rickdenatale](http://www.linkedin.com/in/rickdenatale)