[ANN] testy.rb - ruby testing that's mad at the world

NAME
   testy.rb

DESCRIPTION
   a BDD testing framework for ruby that's mad at the world and plans to kick
   it's ass in 80 freakin lines of code

SYNOPSIS
   Testy.testing 'your code' do
     test 'some behaviour' do |result|
       ultimate = Ultimate.new
       result.check :name, :expect => 42, :actual => ultimate.answer
     end
   end

PRINCIPLES AND GOALS
   . it.should.not.merely.be.a.unit.testing.with.a.clever.dsl

   . testing should not require learning a framework. ruby is a great
   framework so testy uses it instead, requiring programmers learn exactly 2
   new method calls

   . testing loc should not dwarf those of the application

   . testing framework loc should not dwarf those of the application

   . testing frameworks should *never* alter ruby built-ins nor add methods to
   Object, Kernel, .et al

   . the output of tests should be machine parsable for reporting and ci tools
   to easily integrate with

   . the output of tests should be beautiful so that humans can read it

   . the shape of the test file should not insult the programmer so that tests
   can double as sample code

   . the testing framework should never alter exception semantics

   . hi-jacking at_exit sucks ass

   . the exit status of running a test suite should indicate the degree of it's
   failure state: the more failures the higher the exit status

   . sample code should easily be able to double as a test suite, including
   it's output

   . testing should improve your code and help your users, not make you want to
   kill yourself

   . using a format that aligns in terminal is sanity saving when comparing
   output

   . testing frameworks should provide as few shortcuts for making brittle
   tightly coupled tests as possible

   . test suites should be able to be created, manipulated, have their output
   streamed to different ports, and even tested themselves - they should be
   plain ol objects under the hood

SAMPLES

   <========< samples/a.rb >========>

   ~ > cat samples/a.rb

     # simple use of testy involves simply writing code, and recording the result
     # you expect against the actual result

···

#
     # notice that the output is very pretty and that the exitstatus is 0 when all
     # tests pass
     #
       require 'testy'

       Testy.testing 'the kick-ass-ed-ness of testy' do

         test 'the ultimate answer to life' do |result|
           list = []

           list << 42
           result.check :a, :expect => 42, :actual => list.first

           list << 42.0
           result.check :b, 42.0, list.last
         end

       end

   ~ > ruby samples/a.rb #=> exitstatus=0

     ---
     the kick-ass-ed-ness of testy:
       the ultimate answer to life:
         success:
           a: 42
           b: 42.0

   <========< samples/b.rb >========>

   ~ > cat samples/b.rb

     # testy will handle unexpected results and exceptions thrown in your code in
     # exactly the same way - by reporting on them in a beautiful fashion and
     # continuing to run other tests. notice, however, that an unexpected result
     # or raised exception will cause a non-zero exitstatus (equalling the number
     # of failed tests) for the suite as a whole. also note that previously
     # accumulate expect/actual pairs are still reported on in the error report.
     #
       require 'testy'

       Testy.testing 'the exception handling of testy' do

         test 'raising an exception' do |result|
           list = []

           list << 42
           result.check :a, :expect => 42, :actual => list.first

           list.method_that_does_not_exist
         end

         test 'returning unexpected results' do |result|
           result.check 'a', 42, 42
           result.check :b, :expect => 'forty-two', :actual => 42.0
         end

       end

   ~ > ruby samples/b.rb #=> exitstatus=2

     ---
     the exception handling of testy:
       raising an exception:
         failure:
           error:
             class: NoMethodError
             message: undefined method `method_that_does_not_exist' for [42]:Array
             backtrace:
             - samples/b.rb:18
             - ./lib/testy.rb:65:in `call'
             - ./lib/testy.rb:65:in `run'
             - /opt/local/lib/ruby/site_ruby/1.8/orderedhash.rb:65:in `each'
             - /opt/local/lib/ruby/site_ruby/1.8/orderedhash.rb:65:in `each'
             - ./lib/testy.rb:61:in `run'
             - ./lib/testy.rb:89:in `testing'
             - samples/b.rb:10
           expect:
             a: 42
           actual:
             a: 42
       returning unexpected results:
         failure:
           expect:
             a: 42
             b: forty-two
           actual:
             a: 42
             b: 42.0

a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama

You call this beautiful, but I don't understand it. This says that 'a'
is okay and 'b' isn't, right? Maybe it's not so much that I don't
understand it as I don't really like it.

Frankly, I find it rather ironic that you're writing a testing
framework and seemingly advocating BDD. Maybe things have changed
mightily in these heady recent times.

I like some of what you have as points, like the output should be
readable ("beautiful" is a little subjective), and of course that
tests should improve your code. The framework points, about the
framework not being huge and not contributing to brittle tests are
good, and the exit status is interesting. Personally, I live with a
couple of methods (as few as possible) on Object and Kernel so writing
the tests doesn't make me want to kill myself.

I used RSpec for a long time, and still do with some projects. I've
switched bacon for my personal projects, and I love it. As for
mocking, which is necessary in some cases if you want to test without
turning into a serial killer, mocha with RSpec, facon with bacon.

···

On Mar 28, 6:01 pm, "ara.t.howard" <ara.t.how...@gmail.com> wrote:

   returning unexpected results:
     failure:
       expect:
         a: 42
         b: forty\-two
       actual:
         a: 42
         b: 42\.0

--
-yossef

ara.t.howard wrote:

   . it.should.not.merely.be.a.unit.testing.with.a.clever.dsl

How about you simply let the programmer write anything they want, and then if it returns false or nil you rip their entire expression and report the name and value of every variable within it?

   assert{ foo > 41.8 and foo < 42.1 } => false

      foo => 40.9

Notice we didn't need to tell the assertion the variable's name was 'foo'. This rewards programmers for writing readable code. You get the maximum reflection for the leanest statements.

Honestly I think it's _lack_ of a nauseatingly kewt DSL that inhibits adoption of assert{ 2.0 }...

ara.t.howard wrote:

NAME
   testy.rb

DESCRIPTION
   a BDD testing framework for ruby that's mad at the world and plans to kick it's ass in 80 freakin lines of code

It's nice to see you finally riffing on testing.

Later,

···

--
http://twitter.com/bil_kleb
http://fun3d.larc.nasa.gov

Up to a limit of course. how about exiting with the percentage ? Exit status
is limited to 256 values, so you can make it exit 0 with lots of failures:

    gist:87480 · GitHub

enjoy,

-jeremy

···

On Sun, Mar 29, 2009 at 08:01:18AM +0900, ara.t.howard wrote:

  . the exit status of running a test suite should indicate the degree of
    it's failure state: the more failures the higher the exit status

--

Jeremy Hinegardner jeremy@hinegardner.org

Ara Howard wrote:

       result.check :name, :expect => 42, :actual => ultimate.answer

I'm afraid I'm missing something. Why is this better than

  assert_equal 42, ultimate.answer, "name"
?

   . testing should improve your code and help your users, not make
you want to
   kill yourself

Hear hear to that!

requiring programmers learn exactly 2 new method calls

Well, it would be nice to know what those 2 methods calls were, and
their semantics, without reverse-engineering the code. Are these
Testy.testing and Result#check ?

I did find the gem, and installed it, but the generated rdoc is entirely
free of comments or explanation.

The two samples don't really help me understand why testy is good or how
to use it effectively, since they are both more verbose that what I
would have written in test/unit to get the same result.

How would I do something equivalent to these?

   assert_raises(RuntimeError) { somecode }

   assert_match /error/i, response.body

I think I'd also miss the ability to have setup and teardown before each
test (something which 'shoulda' makes very simple and effective).

Please don't get me wrong - I'm absolutely interested in something which
will make testing simpler and easier, if I can understand how to use it
effectively.

Regards,

Brian.

···

--
Posted via http://www.ruby-forum.com/\.

Ara Howard wrote:

NAME
   testy.rb

DESCRIPTION
   a BDD testing framework for ruby that's mad at the world and plans
to kick
   it's ass in 80 freakin lines of code

SYNOPSIS
   Testy.testing 'your code' do
     test 'some behaviour' do |result|
       ultimate = Ultimate.new
       result.check :name, :expect => 42, :actual => ultimate.answer
     end
   end

Hey Ara,

Interesting post and project. Nice work! I just tried to test testy.rb
out and, maybe i'm overlooking something obvious, but when i run:

ruby testy_test.rb (http://pastie.org/430735\)

Desktop$: ruby testy_testing.rb

···

---
naming a student:
   retrieving a student's name:
     failure:
       expect:
         name:
       actual:
         name: Lake
   giving a student a name:
     failure:
       expect:
         name:
      actual:
        name: Jake

You see, the output does not contain the second Testy.testing()
results...

What gives?

Thanks,

Lake
--
Posted via http://www.ruby-forum.com/\.

IIRC, PragDave suggested the at_exit trick to Talbott during
the elder days.

Later,

···

On Mar 28, 7:01 pm, "ara.t.howard" <ara.t.how...@gmail.com> wrote:

. hi-jacking at_exit sucks ass

--
http://twitter.com/bil_kleb

Yossef Mendelssohn wrote:

You call this beautiful, but I don't understand it. This says that 'a'
is okay and 'b' isn't, right? Maybe it's not so much that I don't
understand it as I don't really like it.

Actually, it seems to be YAML format. It's readable and can be parsed.

Regards,
Ian

···

--
Posted via http://www.ruby-forum.com/\.

       returning unexpected results:
         failure:
           expect:
             a: 42
             b: forty-two
           actual:
             a: 42
             b: 42.0

You call this beautiful, but I don't understand it. This says that 'a'
is okay and 'b' isn't, right? Maybe it's not so much that I don't
understand it as I don't really like it.

it's a valid complaint. but compare it to what you'll get in most frameworks and consider that, by beautiful, i mean that a YAML.load can slurp the entire set of expect vs actual. i considered a delta style format:

   diff
     a:
       expect: 42
       actual: 42.0
     b:
       expect: 43
       actual: 43.0

but it seems very hard to reconstruct for downstream filters. i'm open to suggestion on format though. requirements are

  . readable by humans
         . easily parsed by computers

basically that means some yaml format. honestly open to suggestion here...

Frankly, I find it rather ironic that you're writing a testing
framework and seemingly advocating BDD. Maybe things have changed
mightily in these heady recent times.

i personally don't think so, i think the community took a wrong turn, from wikipedia (http://en.wikipedia.org/wiki/Behavior_Driven_Development\)

"
The practices of BDD include:

     * Involving stakeholders in the process through outside-in software development

     * Using examples to describe the behavior of the application, or of units of code
     * Automating those examples to provide quick feedback and regression testing

     * In software tests, using 'should' to help clarify responsibility and allow the software's functionality to be questioned
     * Test use 'ensure' to differentiate outcomes in the scope of the code in question from side-effects of other elements of code.
     * Using mocks to stand-in for modules of code which have not yet been written

i have major issues with points two and three wrst to most ruby testing frameworks. one of the main points of testy is to combine examples with testing. rspec and all the others do not serve as examples unless you are a ruby master. that is to say they introduce too many additions to the code that's supposed to be an example to really preserve it's 'exampleness'. and of course the output is utterly useless to normal humans. if a framework provides 1000 asset_xxxxxxxx methods ad nausea then the point of the code - it's level of example-good-ness - is lost to mere mortals

I like some of what you have as points, like the output should be
readable ("beautiful" is a little subjective), and of course that
tests should improve your code. The framework points, about the
framework not being huge and not contributing to brittle tests are
good, and the exit status is interesting. Personally, I live with a
couple of methods (as few as possible) on Object and Kernel so writing
the tests doesn't make me want to kill myself.

I used RSpec for a long time, and still do with some projects. I've
switched bacon for my personal projects, and I love it. As for
mocking, which is necessary in some cases if you want to test without
turning into a serial killer, mocha with RSpec, facon with bacon.

this will summarize where my thoughts are on that

cfp:~/redfission > find vendor/gems/{faker,mocha,thoughtbot}* -type f|xargs -n1 cat|wc -l
    24255

cfp:~/redfission > find app -type f|xargs -n1 cat|wc -l
     1828

rspec and co might be fine but seriously, the above is insane right?

kind regards.

a @ http://codeforpeople.com/

···

On Mar 28, 2009, at 5:33 PM, Yossef Mendelssohn wrote:

On Mar 28, 6:01 pm, "ara.t.howard" <ara.t.how...@gmail.com> wrote:

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama

that's interesting indeed. one of my goals with testy is that output is meaningful both for computers and humans and, for that, yaml is tops.

still - reporting on the context the errors was thown raised from is quite interesting. you are basically saying report binding not stacktrace right?

a @ http://codeforpeople.com/

···

On Mar 28, 2009, at 6:00 PM, Phlip wrote:

ara.t.howard wrote:

  . it.should.not.merely.be.a.unit.testing.with.a.clever.dsl

How about you simply let the programmer write anything they want, and then if it returns false or nil you rip their entire expression and report the name and value of every variable within it?

assert{ foo > 41.8 and foo < 42.1 } => false

    foo => 40.9

Notice we didn't need to tell the assertion the variable's name was 'foo'. This rewards programmers for writing readable code. You get the maximum reflection for the leanest statements.

Honestly I think it's _lack_ of a nauseatingly kewt DSL that inhibits adoption of assert{ 2.0 }...

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama

:wink: more to come

a @ http://codeforpeople.com/

···

On Mar 28, 2009, at 9:30 PM, Bil Kleb wrote:

It's nice to see you finally riffing on testing.

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama

Jeremy Hinegardner wrote:

···

On Sun, Mar 29, 2009 at 08:01:18AM +0900, ara.t.howard wrote:

  . the exit status of running a test suite should indicate the degree of it's failure state: the more failures the higher the exit status

Up to a limit of course. how about exiting with the percentage ? Exit status
is limited to 256 values, so you can make it exit 0 with lots of failures:

Again: If you have any reason to _count_ the errors, you are already absolutely screwed anyway...

Brian Candler wrote:

Ara Howard wrote:

       result.check :name, :expect => 42, :actual => ultimate.answer

I'm afraid I'm missing something. Why is this better than

  assert_equal 42, ultimate.answer, "name"
?

Or even less:

   name = 42
   assert{ name == ultimate.answer }

good catch - that was uh, tired of me :wink:

i'll add precent now.

a @ http://codeforpeople.com/

···

On Mar 29, 2009, at 12:16 PM, Jeremy Hinegardner wrote:

Up to a limit of course. how about exiting with the percentage ? Exit status
is limited to 256 values, so you can make it exit 0 with lots of failures:

   gist:87480 · GitHub

enjoy,

-jeremy

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama

Ara Howard wrote:

      result.check :name, :expect => 42, :actual => ultimate.answer

I'm afraid I'm missing something. Why is this better than

assert_equal 42, ultimate.answer, "name"

you can basically do that too, but i continually forget which is expected and which is actual and, as you know, that's a slippery error to track down at times.

  . testing should improve your code and help your users, not make
you want to
  kill yourself

Hear hear to that!

:wink:

requiring programmers learn exactly 2 new method calls

Well, it would be nice to know what those 2 methods calls were, and
their semantics, without reverse-engineering the code. Are these
Testy.testing and Result#check ?

yes. that is it.

I did find the gem, and installed it, but the generated rdoc is entirely
free of comments or explanation.

what are these comments you speak of?

seriously, this is an experiment at this point but they will be forthcoming if it sprouts wings

The two samples don't really help me understand why testy is good or how
to use it effectively, since they are both more verbose that what I
would have written in test/unit to get the same result.

How would I do something equivalent to these?

  assert_raises(RuntimeError) { somecode }

  assert_match /error/i, response.body

at this point my feeling is that breaks ths concept of BDD

"Using examples to describe the behavior of the application, or of units of code"

because littering the example code with esoteric testing framework voodoo turns it into code in the testing language that does not resemble how people might actually use the code - at least not without mentally unraveling considerable indirection. i always end up writing both samples and tests - one of the goals of testy is that, by having a really simple interface and really simple human friendly output we can just write examples that double as tests. in the latest testy you can do this

cfp:~/src/git/testy > cat a.rb
require 'testy'

Testy.testing 'my lib' do

   test 'foo' do |result|
     list = [42]
     result.check :fooness, :expect => 42, :actual => list.last
   end

   test 'bar' do |result|
     list = [42.0]
     result.check :barness, :expect => 42.0, :actual => list.last
   end

end

get a listing of which tests/examples i can run

cfp:~/src/git/testy > ruby -I lib a.rb --list

···

On Mar 29, 2009, at 1:51 PM, Brian Candler wrote:
---
- foo
- bar

run one of them (actually regex matching so you can select more that one)

cfp:~/src/git/testy > ruby -I lib a.rb bar
---
my lib:
   bar:
     success:
       barness: 42.0

you can also do something like this (experimental) to just make a simple example

cfp:~/src/git/testy > cat a.rb
require 'testy'

Testy.testing 'my lib' do

   test 'just an example of summing an array using inject' do
     a = 1,2
     a.push 3
     sum = a.inject(0){|n,i| n += i}
   end

end

cfp:~/src/git/testy > ruby -I lib a.rb 'just an example'
---
my lib:
   just an example of summing an array using inject:
     success: 6

testy will just display the return value if no results are explicitly checked but, of course, exceptions are still reported and cause a failed test in the normal way.

so the goal is making it even easier to have a user play with your tests/examples to see how they work, and even to allow simple examples to be integrated with your test suite so you make sure you samples still run without error too. of course you can do this with test/unit or rspec but the output isnt' friendly in the least - not from the perspective of a user trying to learn a library, nor is it useful to computers because it cannot be parsed - basically it's just vomiting stats and backtraces to the console that are hard for people to read and hard for computers to read. surely i am not the only one that sometimes resorts to factoring out a failing test in a separate program because test/unit and rspec output is too messy to play nice with instrumenting code? and that's not even getting to what they do with at_exit exception raising...

I think I'd also miss the ability to have setup and teardown before each
test (something which 'shoulda' makes very simple and effective).

yeah that's on deck for sure. i *do* really like contexts with shoulda. but still

cfp:/opt/local/lib/ruby/gems/1.8/gems/thoughtbot-shoulda-2.9.1 > find lib/ -type f|xargs -n1 cat|wc -l
     3910

if we accept the research and assume that bugs scale linerarly with the # of lines of code this is not good for robustness. this is one of my main gripes with current ruby testing - my current rails app has about 1000 lines of code and 25,000 lines of testing framework!

Please don't get me wrong - I'm absolutely interested in something which
will make testing simpler and easier, if I can understand how to use it
effectively.

feedback is awesome - this is an new idea i'm just fleshing out so i really appreciate it.

cheers.

a @ http://codeforpeople.com/
--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama

my stupidity. i hadn't considered having more that one in a file :wink: actually a bit tricky with the exit status issue - but i'll roll out a fix.

/me hangs head

a @ http://codeforpeople.com/

···

On Mar 29, 2009, at 2:44 PM, Lake Denman wrote:

You see, the output does not contain the second Testy.testing()

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama

bingo. emphasis on the latter. think unix pipes.

a @ http://codeforpeople.com/

···

On Mar 28, 2009, at 5:45 PM, Ian Trudel wrote:

Actually, it seems to be YAML format. It's readable and can be parsed.

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama

I really feel counting the errors and reading the output are both things better handled by defining a good interface for the results writer. If I could just define some trivial class with methods like test_passed(), test_failed(), test_errored_out(), and tests_finished() then just plug that in, I could easily do anything I want.

James Edward Gray II

···

On Mar 29, 2009, at 1:39 PM, Phlip wrote:

Jeremy Hinegardner wrote:

On Sun, Mar 29, 2009 at 08:01:18AM +0900, ara.t.howard wrote:

. the exit status of running a test suite should indicate the degree of it's failure state: the more failures the higher the exit status

Up to a limit of course. how about exiting with the percentage ? Exit status
is limited to 256 values, so you can make it exit 0 with lots of failures:

Again: If you have any reason to _count_ the errors, you are already absolutely screwed anyway...

indeed. i was vaugely thinking of a status report with failure reported by severity. really just an idea at this point.

a @ http://codeforpeople.com/

···

On Mar 29, 2009, at 12:39 PM, Phlip wrote:

Again: If you have any reason to _count_ the errors, you are already absolutely screwed anyway...

--
we can deny everything, except that we have the possibility of being better. simply reflect on that.
h.h. the 14th dalai lama