# Minitest randomization **URL:** \ **Category:** ruby-talk\ **Created:** [10 October 2010 04:38 UTC](https://rubytalk.org/t/minitest-randomization/60387 "2010-10-10T04:38:50Z")\ **Posts on this page:** 1\ **Showing post:** 3
**Author:** ![Alex\_Chaffee1](https://avatars.discourse-cdn.com/v4/letter/a/f14d63/32.png) [@Alex\_Chaffee1](https://rubytalk.org/u/Alex_Chaffee1)\ **Post date:** [11 October 2010 03:33 UTC](https://rubytalk.org/t/minitest-randomization/60387/3 "2010-10-11T03:33:44Z")
(Apologies if the quotes don't come out right in plain text -- I'm using both Apple Mail and GMail and they're playing crazy HTML games with my draft.) > > I guess this comes down to idempotency. I expect that if I do > > something twice in a row I will get the same result. Randomizing by > > > > > default breaks this expectation. It's astonishing, therefore bad, no > > > > > matter how good from a theoretical standpoint, and especially > > > > > astonishing when people have 10+ years of xUnit and its heirs building > > > > > these expectations. > > > > > Idempotency is a red-herring. There is nothing about the xunit > family/philosophy of tools (or any other test tool that I have used or > studied--except rspec) that suggests that test order must be run in the > order defined (or must be run in any order at all). Just look at the new > tools coming out that distribute and multithread/process your tests and you > can see that right there, the notion has to be thrown out the window by > design. That's a fair point. The idempotency I was referring to was that running "rake test" twice on a failing suite gets different results -- if not different failures, then the same failures in a different order. As for your astonishment, I thought it was pretty well addressed in the > first line of my reply: "Really? I think preventing test order dependency > has a very practical effect". If you're still astonished after that, then > you're probably misusing the word. I'm using it in its technical sense: > **[Principle of least astonishment](https://en.wikipedia.org/wiki/Principle_of_least_astonishment)** > > In user interface design and software design, > the principle of least astonishment (POLA), also known as principle of least surprise,\[a\] proposes that a component of a system should behave in a way that most users will expect it to behave, and therefore not astonish or surprise users. The following is a corollary of the principle: "If a necessary feature has a high astonishment factor, it may be necessary to redesign the feature." > The principle has been in use in relation to computer interaction... And I stand by what I wrote: if your tests are all passing, and they're well isolated, then randomizing them has no practical effect. It's just shuffling a deck full of aces. At this point I'm going to cut much of your reply and everything I've > written so far in response and cut to the chase: And I'm not even disagreeing with your observation! I agree that there should be a randomizing mode, and that people should run it fairly often. Just not all the time and not without a config or command-line option to turn it off. Apparently this is the crux of our disagreement: > I \_\_do\_\_ think that people should randomize their tests a MAJORITY of the > time and turn it off TEMPORARILY when they need to sort out an issue. If it > wasn't random by default, it wouldn't happen at all. This is a noble position, as I said before. You're the self-appointed isolation vigilante, crusading against a problem you abhor. But I've rarely encountered it. I feel that my tests don't need randomization, and the extra output clutters my console (\*), and the shuffling cramps my debugging style, so I want it off unless I ask for it. If you're Batman, I feel like I'm the Lorax. I speak for the trees whose pristine consoles are being polluted, but who haven't spoken out. (I haven't really heard a chorus of protestors in favor of randomization either, fwiw.) You're the library author, so you have the privilege of deciding what mode is the default. I'm hoping to convince you of a few things, but if I don't, I won't take it personally. Sounds like we're approaching a compromise, though: an option for me, defaulting to off for you. (And option != monkey patch -- it's a clear API like a named switch on the command line and/or a value on some Minitest object, e.g. "Minitest::Config.randomize = false".) I'd also be happy with just a verbosity setting, maybe with several levels like you suggest. 2) use --seed when you want the order to be fixed, via TESTOPTS if you're > using rake. And that'd be your command line option... TESTOPTS. Roger that. Never used it before. Maybe the Minitest README should say something about that when it talks about --seed. (Oh, looks like it doesn't talk about --seed either.) If you want a third option available, feel free to propose it and I'll > gladly consider it. Had a weird thought while doing the dishes... what if you write out seed somewhere persistent like .minitest\_seed, then erase it after the run but only if the run was successful. Then when a run starts, if .minitest\_seed exists, it uses it (and says so) instead of rolling a new one. That way you don't have to print out anything for successful runs and the user doesn't have to remember anything and idempotency is preserved (if it fails once, it'll fail the next time, in exactly the same way) and it'll keep failing consistently until you fix the problem. It also works for C hackers since a crash means it won't erase the cached seed. (And hey, also, for Aaron's sake, can't you trap SIGSEGV and print the seed then? Not a rhetorical question since I haven't done any C+Ruby stuff and I know signals are sometimes flakey.) Since --seed makes in-test randomization freeze out too, I think there should be separate options for all three (--seed, --randomize, and --verbose). P.S. I'm still mulling over Steve Klabnik's suggestion that the output be > sorted. I think it could be very confusing when you do have test dependency > errors but that there might be some way to mitigate the confusion. I'd like > to hear what you think about his suggestion. I like it. It's pretty weird though. it's a very pleasant dream I'm not sure if it will survive in the cold light of day. - A (\*) My conosle is already way cluttered even with the minimum verbosity -- my collaborator Steve wrote some code that runs each of our tests in its own VM process, to ensure isolation of dependencies and other stuff, so I get a big long scroll of test runs, each of which is now 2 lines longer because of "test run output" cruft. git clone wrong and run "rake rvm:test" to see what I mean. Every line of output I save is multiplied by (N tests) x (M Ruby versions). Since it's slow, I only run it before checkin. --- _[View the full topic](https://rubytalk.org/t/minitest-randomization/60387)._