[Ann] Verify, a very basic testing tool

ruby-talk
1

Hi List

The Verify tool is a KISS verification and test tool, it is inspired
by Ara T. Howard's testy.rb.
Ara however has still implemented a BDD framework, the purpose of
Verify is to provide a Ruby Programmer
with a very simple yet effective tool to test her assumptions about code.

Verify was motivated by a reply to Ara's release post made by Phlip.
Phlip has made an excellent point about assert.
And that very often that is (almost?) all what is needed.
The Verify tool tries to implement exactly that idea and is the
simplest possible tool to do the job (for me).

Verify has the following goals

   * Almost nothing to learn, one wrapper method and three test
methods with almost no parameters all save one
     optional.
      # At the end a resume of the verifications with an error message
for each failed verification
      # will be printed to stderr or a mocking object.
      # Strings and Arrays are fine for that purpose, as the reporter
uses #<< exclusively.

   * No metaprogramming, no at_exit hook, execution of the
verification inside a wrapper object, no global namespace pollution,
     except the Verify method.

   * Wrapper behavior inference to the testee is limited to #verify,
#verify_not and #verify_exceptions.

   * Very simple output

   * "Last Line Says It All" feature "================" --> all tests
ok, "***************" --> not all tests ok.

USAGE EXAMPLES:

        require 'verify'

        Verify do
          verify do 42 end
          verify_not do nil end
          verify_exceptions NameError do abcdefghijk end
        end

        Verify "that the universe is ok" do
          verify "this only shows in error reports" do
            1 / 0
          end
        end

LICENCE:
        BSD

Enjoy
Robert

Ah yeah BTW if someone really wants to download this :wink:

http://rubyforge.org/frs/?group_id=3824&release_id=30476

···

--
There are some people who begin the Zoo at the beginning, called
WAYIN, and walk as quickly as they can past every cage until they get
to the one called WAYOUT, but the nicest people go straight to the
animal they love the most, and stay there. ~ A.A. Milne (from
Winnie-the-Pooh)

2

refute: same length as verify!
raises: ditto!
(assert: I like assert)

···

On Apr 3, 2009, at 13:02 , Robert Dober wrote:

       Verify do
         verify do 42 end
         verify_not do nil end
         verify_exceptions NameError do abcdefghijk end
       end

3

Without proper announce, I'd like to show that I had something similar
for quite some time now, as part of my knock project:

% cat t.tb
require 'knock'
include Knock

testing "various expectances" do
  expect(42) { 6 * 7 }
  expect(nil) { nil }
  expect(NameError) { abcdefghijk }

  is 42, 6*7
  ok 42 == 6*7
end

testing "that the universe is ok" do
  testing "this only shows in error reports" do
    1 / 0
  end
end

% ruby t.rb
ok - various expectances - expecting value 42
ok - various expectances - expecting value nil
ok - various expectances - expecting exception NameError
ok - various expectances
ok - various expectances
not ok - /Users/chris/mess/current/t.rb:14 - that the universe is ok - this only shows in error reports - RAISED: divided by 0
# /Users/chris/mess/current/t.rb:15:in `/'
# /Users/chris/mess/current/t.rb:15
# /Users/chris/mess/current/t.rb:14
# /Users/chris/mess/current/t.rb:13

% ruby t.rb | kn-agg
not ok - 6 tests, 1 failed (83.3333% succeeded)

The expect API is a rip-off from http://expectations.rubyforge.org/,
but simplified to its core:

% wc -l knock.rb kn-agg
      61 knock.rb
      10 kn-agg

Sources can be found at

There are implementations for C, Ruby, sh/bash/zsh, and it is trivial
to implement in any language that supports "puts" in some way.

···

--
Christian Neukirchen <chneukirchen@gmail.com> http://chneukirchen.org

4

Not to be contentious, but you use both define_method and *_eval to
generate code. Unless I don't know what that term means, I believe
you *do* use metaprogramming.

Though, I do have a question/concern/confusion/something. I
completely and utterly fail to see the point of these new closure
obsessed frameworks or tools. Nearly every bit of their functionality
could be wrapped up in a few simple helpers like this:

class Test::Unit::TestCase
  alias old_assert assert

  def assert(result = nil, &block)
    result = block.call if block_given?
    old_assert result
  end
end

Unless I'm totally missing something? I realize it's a different
syntax or "DSL," and that's fine if that's your reasoning, but I don't
see how they're all that revolutionary. I like closures as much as
the next guy, but I don't see how they revolutionize my testing life.

</rant>

Aimed much less at you, and more at the 4-5 of these I've seen in the
past few days.

--Jeremy

···

On Fri, Apr 3, 2009 at 3:02 PM, Robert Dober <robert.dober@gmail.com> wrote:

Hi List

The Verify tool is a KISS verification and test tool, it is inspired
by Ara T. Howard's testy.rb.
Ara however has still implemented a BDD framework, the purpose of
Verify is to provide a Ruby Programmer
with a very simple yet effective tool to test her assumptions about code.

Verify was motivated by a reply to Ara's release post made by Phlip.
Phlip has made an excellent point about assert.
And that very often that is (almost?) all what is needed.
The Verify tool tries to implement exactly that idea and is the
simplest possible tool to do the job (for me).

Verify has the following goals

* Almost nothing to learn, one wrapper method and three test
methods with almost no parameters all save one
optional.
# At the end a resume of the verifications with an error message
for each failed verification
# will be printed to stderr or a mocking object.
# Strings and Arrays are fine for that purpose, as the reporter
uses #<< exclusively.

* No metaprogramming, no at_exit hook, execution of the
verification inside a wrapper object, no global namespace pollution,
except the Verify method.

* Wrapper behavior inference to the testee is limited to #verify,
#verify_not and #verify_exceptions.

* Very simple output

* "Last Line Says It All" feature "================" --> all tests
ok, "***************" --> not all tests ok.

USAGE EXAMPLES:

   require &#39;verify&#39;

   Verify do
     verify do 42 end
     verify\_not do nil end
     verify\_exceptions NameError do abcdefghijk end
   end

   Verify &quot;that the universe is ok&quot; do
     verify &quot;this only shows in error reports&quot; do
       1 / 0
     end
   end

LICENCE:
BSD

Enjoy
Robert

Ah yeah BTW if someone really wants to download this :wink:

http://rubyforge.org/frs/?group_id=3824&release_id=30476

--
There are some people who begin the Zoo at the beginning, called
WAYIN, and walk as quickly as they can past every cage until they get
to the one called WAYOUT, but the nicest people go straight to the
animal they love the most, and stay there. ~ A.A. Milne (from
Winnie-the-Pooh)

--
http://jeremymcanally.com/
http://entp.com/

My books:

http://humblelittlerubybook.com/ (FREE!)

5 
  Verify do
    verify do 42 end
    verify\_not do nil end
    verify\_exceptions NameError do abcdefghijk end
  end

refute: same length as verify!

Never heard of that word before, just checked it on Websters, thanx
Ryan great idea.

raises: ditto!
(assert: I like assert)

So do I, but assert is normally inserted into production code, and
Verify is still a testing tool, but you know it is so easy to alias
the methods in the anonymous module that creates the wrapper object in
the Verify method ;).
Cheers
Robert

···

On Fri, Apr 3, 2009 at 10:56 PM, Ryan Davis <ryand-ruby@zenspider.com> wrote:

On Apr 3, 2009, at 13:02 , Robert Dober wrote:

--
There are some people who begin the Zoo at the beginning, called
WAYIN, and walk as quickly as they can past every cage until they get
to the one called WAYOUT, but the nicest people go straight to the
animal they love the most, and stay there. ~ A.A. Milne (from
Winnie-the-Pooh)

6

Jeremy McAnally wrote:

Though, I do have a question/concern/confusion/something. I
completely and utterly fail to see the point of these new closure
obsessed frameworks or tools.

Expensive setup is a design smell.

That said, sometimes your setups are expensive. Consider this outline:

   describe 'frob_controller' do

     setup do
       @p_frobs = assemble_some_paid_up_frobs()
     end

     specify ... specify ... specify 'behaviors of paid-up frobs'

     describe 'delinquent frobs'
       setup do
         @d_frobs = assemble_some_delinquent_frobs()
       end

       specify 'how paid-up and delinquent frobs interact' do
         mess_with_both @p_frobs, @d_frobs
       end
     end

   end

The first setup builds ideal frobs, and the specify blocks (elided) test them.

The second setup is nested, so its specifications run with both setups. The nested specify blocks can use both kinds of frobs, together. You get this by directly illustrating how the concepts nest. You don't need to write each setup once and then somehow re-use it into various test suites.

A 'def test_frob' is a legacy of Java and its block-free programming styles. Yet a test case is _not_ a method. You should not want to call it.

Treating blocks of testing code as objects makes weaving them together easier. Contrast the "abstract test pattern", which Test::Unit::TestCase makes an absolute pain in the 'nads. It's probably just as much fun in Java...

···

--
   Phlip

7

<snip>

Though, I do have a question/concern/confusion/something. I
completely and utterly fail to see the point of these new closure
obsessed frameworks or tools.

I do not speak for everyone, but using closures just seems to be
right, it hides the data, avoids any possible clash and is very
elegant, but that is my personal view.
If you do not like closures it will be difficult to understand those
who like them, unless you undergo the trouble to work with closures
yourself and decide if it gives you an edge or not.
Furthermore I believe that using closures allows you to look a little
but farther than Ruby, which might be a good thing.
<snip>
I like closures as much as

the next guy, but I don't see how they revolutionize my testing life.

I would not think that this has anything to do with testing. All I
want to achieve is to minimize the risk that the user of a framework
stumbles into a debugging nightmare because she stepped on the feet of
the framework.

</rant>

Oh that was a rant? I thought it was a very sensible question and I am
not cynical here at all.
Did I waste my time?

Aimed much less at you, and more at the 4-5 of these I've seen in the
past few days.

No offense taken ... bones :wink:
R.

···

On Sat, Apr 4, 2009 at 7:35 PM, Jeremy McAnally <jeremymcanally@gmail.com> wrote:

8

Thanks to Ryan I have changed #verify_not to #refute.
But not #alert and #raises because I am afraid that these method names
might shadow too much testee code :(.

Cheers
Robert

http://rubyforge.org/frs/?group_id=3824 (verify-0.1a)

9

Robert Dober wrote:

···

On Fri, Apr 3, 2009 at 10:56 PM, Ryan Davis <ryand-ruby@zenspider.com> wrote:

On Apr 3, 2009, at 13:02 , Robert Dober wrote:

      Verify do
        verify do 42 end
        verify_not do nil end
        verify_exceptions NameError do abcdefghijk end
      end

refute: same length as verify!

Never heard of that word before, just checked it on Websters, thanx
Ryan great idea.

I still contend that "refute" means to prove something is not true, not to prove it false. refute do false end is meaningless.

- Charlie

10

Robert Dober wrote:

      Verify do
        verify do 42 end
        verify_not do nil end
        verify_exceptions NameError do abcdefghijk end
      end

refute: same length as verify!

Never heard of that word before, just checked it on Websters, thanx
Ryan great idea.

Drat, I didn't explain it well, and a quick check shows that refute is "to disprove or to prove false", according to most dictionaries. I guess I just don't like it as an opposite of "assert". All the thesauruses I've checked list the antonyms of "assert" as "deny" and "reject", and the antonyms of "refute" as "demonstrate", "prove", "accept", or "embrace". There's a disconnect there I can't seem to remedy.

- Charlie

···

On Fri, Apr 3, 2009 at 10:56 PM, Ryan Davis <ryand-ruby@zenspider.com> wrote:

On Apr 3, 2009, at 13:02 , Robert Dober wrote:

11

Then go use an instance based inheritance language like self. Classes work just fine and aren't expensive at all for me.

···

On Apr 4, 2009, at 19:14 , Phlip wrote:

Jeremy McAnally wrote:

Though, I do have a question/concern/confusion/something. I
completely and utterly fail to see the point of these new closure
obsessed frameworks or tools.

Expensive setup is a design smell.

12

oops I forgot something important
Verify is Ruby1.9 only, do not look back ;).
R.

13

I agree with you that 'refute' doesn't seem right. Not wanting to get
too pedantic about all this (but failing miserably :), from
http://www.abdn.ac.uk/philosophy/guide/glossary.shtml:

"REFUTE To refute a proposition or theory is to establish or prove
that it is false. Lately many people have taken to using ‘refute’ as a
synonym for ‘deny’, but avoid this usage in philosophy. To deny that
God exists is not, in philosophical usage, to refute (or disprove) the
proposition that God exists."

Generally 'I assert P' means 'I am stating that proposition P is
true' and 'I deny Q' means 'I am stating that proposition Q' is false,
which would fit with your interpretation, i.e. if you deny Q and Q
turns out to be false, then you are correct. However, you haven't
proven anything. Your statement and reality just happen to be in
agreement. To refute Q is to ~prove~ that Q is false. There is no
further argument - Q is false and can only be false. I don't see how
we can assert that simply because a test turns out to return a false
value we have proven anything. We are simply getting assurance that
our assertions and reality are in agreement. In short, if we are
looking for a candidate to be the antonym of 'assert', 'deny' seems to
fit the bill.

A further wrinkle: 'prove' itself has the somewhat archaic meaning of
'test'... as in 'the exception that proves the rule' :slight_smile:

Regards,
Sean

···

On Sun, Apr 5, 2009 at 8:11 AM, Charles Oliver Nutter <charles.nutter@sun.com> wrote:

Robert Dober wrote:

On Fri, Apr 3, 2009 at 10:56 PM, Ryan Davis <ryand-ruby@zenspider.com> >> wrote:

On Apr 3, 2009, at 13:02 , Robert Dober wrote:

 Verify do
   verify do 42 end
   verify\_not do nil end
   verify\_exceptions NameError do abcdefghijk end
 end

refute: same length as verify!

Never heard of that word before, just checked it on Websters, thanx
Ryan great idea.

Drat, I didn't explain it well, and a quick check shows that refute is "to
disprove or to prove false", according to most dictionaries. I guess I just
don't like it as an opposite of "assert". All the thesauruses I've checked
list the antonyms of "assert" as "deny" and "reject", and the antonyms of
"refute" as "demonstrate", "prove", "accept", or "embrace". There's a
disconnect there I can't seem to remedy.

- Charlie

14

As I said on ruby-core: You can have your bike-shed for yourself. I don't participate in bike-shed arguments. I have better things to do.

···

On Apr 4, 2009, at 23:57 , Charles Oliver Nutter wrote:

I still contend that "refute" means to prove something is not true, not to prove it false. refute do false end is meaningless.

15

If you English native speakers cannot decide I will make this your problem :wink:
I will rename #verify to
#make_sure_it_behaves_almost_like_that_with_reasonable_certainty and
#refute to #without_wanting_to_offend_anyones_philosophical_and_religious_feelings_i_guess_the_following_might_be_wrong
This will eliminate 99.999999% of name collisions too :wink:

Well I guess you understand what I mean, #refute really seems fine
because we are in a reduced context. As true as it might be that
refute does not prove the non existence of something in real life it
seems completely irrelevant in a boolean logic.
But maybe I should provide

#true? and #false?

BTW. I agree with Phlip, size does matter here, aligned code is just
easier to read.

Robert

There are some people who begin the Zoo at the beginning, called
WAYIN, and walk as quickly as they can past every cage until they get
to the one called WAYOUT, but the nicest people go straight to the
animal they love the most, and stay there. ~ A.A. Milne (from
Winnie-the-Pooh)

16

Robert Dober wrote:

oops I forgot something important
Verify is Ruby1.9 only, do not look back ;).
R.

Damn. I love this sudden burst of creativity around the topic of testing. A Good Thing. Would love to try this new thing of yours. Already successfully using Ara's.

But this 1.9 nonsense frustrates me. Much as I want to dive in (it's installed and ready to go), I simply don't have time to move code I depend upon into an environment where I don't know whether it will fly or not, or whether it will splat after 3 days of investment and use. I wish there was somewhere I could go to find out what gems run with 1.9 and what do not. But, that seems not to be the ruby way. That's plain crazy, say I.

Maybe in a year it'll be different. Meanwhile, 1.87 works for me. Sad.

t.

···

--

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tom Cloyd, MS MA, LMHC - Private practice Psychotherapist
Bellingham, Washington, U.S.A: (360) 920-1226
<< tc@tomcloyd.com >> (email)
<< TomCloyd.com >> (website) << sleightmind.wordpress.com >> (mental health weblog)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

17

Sean O'Halpin wrote:

"REFUTE To refute a proposition or theory is to establish or prove
that it is false. Lately many people have taken to using ‘refute’ as a
synonym for ‘deny’, but avoid this usage in philosophy. To deny that
God exists is not, in philosophical usage, to refute (or disprove) the
proposition that God exists."

You wanna know what sucks? The basic problem here is 'assert_no' is redundant, so we should use one word. Let's try to pick a word.

If that word is 'deny', then you get this:

   assert frobs(rule)
   assert knobs(rule)
   deny clogs(rule)
   assert slobs(rule)

See the problem yet? They don't line up in columns! The eye thinks the third line is something other than an assertion.

So try this:

   assert frobs(rule)
   assert knobs(rule)
   refute clogs(rule)
   assert slobs(rule)

Ah, that's much better! Now you get a gutter in 9th column, and the whole meaning goes into your brain much faster.

Except we can't use "refute" now, despite it exactly parallels 'assert's length, because some language-lawyer wannabe on the interthing found a reason not to!

So try this:

   assert frobs(rule)
   assert knobs(rule)
   denigh clogs(rule)
   assert slobs(rule)

I tried to use El Goog to find some Olde Englische precedent for "denigh". Just my luck nobody used the internet back then...

···

--
   Phlip

18

Sean O'Halpin wrote:

I agree with you that 'refute' doesn't seem right. Not wanting to get
too pedantic about all this (but failing miserably :), from
http://www.abdn.ac.uk/philosophy/guide/glossary.shtml:

Hey, this is exactly my kind of pedantry.

"REFUTE To refute a proposition or theory is to establish or prove
that it is false. Lately many people have taken to using ‘refute’ as a
synonym for ‘deny’, but avoid this usage in philosophy. To deny that
God exists is not, in philosophical usage, to refute (or disprove) the
proposition that God exists."

Generally 'I assert P' means 'I am stating that proposition P is
true' and 'I deny Q' means 'I am stating that proposition Q' is false,
which would fit with your interpretation, i.e. if you deny Q and Q
turns out to be false, then you are correct. However, you haven't
proven anything. Your statement and reality just happen to be in
agreement. To refute Q is to ~prove~ that Q is false. There is no
further argument - Q is false and can only be false. I don't see how
we can assert that simply because a test turns out to return a false
value we have proven anything. We are simply getting assurance that
our assertions and reality are in agreement. In short, if we are
looking for a candidate to be the antonym of 'assert', 'deny' seems to
fit the bill.

A further wrinkle: 'prove' itself has the somewhat archaic meaning of
'test'... as in 'the exception that proves the rule' :slight_smile:

I think that's why I have trouble with it...refute says to me we're disproving something. So we'd be proving something doesn't work, proving something fails, providing something otherwise expected to be true is false. So I'd say that when an assertion fails, that failure successfully refutes the condition being asserted. But refutation being used as assertion seems wrong...we're trying to show a positive result: that some condition returns false or nil. We're not trying to disprove a condition, nor trying to prove a condition expected to be true is actually false. refute implies to me that there's a true expectation we're disproving, rather than an *expected* false/nil result we're trying to prove.

- Charlie

19

Tom Cloyd wrote:

Damn. I love this sudden burst of creativity around the topic of testing. A Good Thing.

Oookay. Here's a sneak preview of assert{ 2.0 } 0.4.8.

You know how Ajax works by generating JavaScript, and slinging it at your web browser? And you know how Rails purportedly tests it with "assert_rjs"? Here's a sample:

     assert_rjs :replace_html, "advanced_filter", ""

Too cute, right?

Wrong! That expands to nothing but a big Regexp, like /Element.update.*advanced_filter/. So a payload of "advanced_filter", or even a subsequent Element.update('advanced_filter'), could fool it.

Further, at work we do a lot of in-house Ajax, so we are at liberty to render entire partials at whim. We require our teeming minions to use only Firefox. But all assert_rjs does with its third argument is drop it into assert_match. That is not powerful enough to constrain our apps!

Just now while writing this post, I got Aaron Paterson's rKelly working in an assert_rjs clone. rKelly uses racc to parse and evaluate JavaScript. This matches our goal of _unit_ testing soft targets. Watir, Selenium, etc. are all great, and they introduced a generation to testing in general. Buuuuuut they work thru the browser. We are not inventing Ajax itself; we just need to accurately spot-check that our own data go into the correct slots in our JavaScript payloads.

So here's a test that simulates a Rails functional test with xhr :get :

     @response = OpenStruct.new(:body => "Element.update(\"label_7\", \"<input
           checked=\\\"checked\\\" id=\\\"Top_Ranking\\\"
           name=\\\"Top_Ranking\\\" type=\\\"checkbox\\\" value=\\\"Y\\\"
           \\/>I want a pet &lt; than a chihuahua<input id=\\\"cross_sale_1\\\"
         name=\\\"cross_sale_1\\\" type=\\\"hidden\\\" value=\\\"7\\\" \\/>\");")

     assert_rjs :replace_html, :label_7

K, so far that looks like the original assert_rjs. But under the hood, it actually lexed the Element.update() call:

     ast.pointcut('Element.update()').matches.each do |updater|
       updater.grep(RKelly::Nodes::ArgumentsNode).each do |thang|
         div_id, html = thang.value

         if target and html
           div_id = eval(div_id.value)
           html = eval(html.value)
           if div_id == target.to_s
             assert_match matcher, html

(Open question to Aaron - is that the best way to run the query?)

The test actually determines we really got hold of the Element.update('label_7', ...). No other JavaScript line will match.

Here's the assertions to match the text payload:

     assert_rjs :replace_html, :label_7, /Top_Ranking/
     assert_rjs :replace_html, :label_7, /pet &lt; than a chihuahua/

Ho hum; so far assert_rjs Classic could have done all that. But...

Because I have an exact string, not a rough match, I can now treat it as pure HTML, and I can drop it into the mighty assert_xhtml()! Now the assertion looks like this:

     assert_rjs :replace_html, :label_7 do
       input.Top_Ranking! :type => :checked, :value => :Y
       input.cross_sale_1, :type => :hidden, :value => 7
     end

From here, no matter how complex that rendered partial, the assertion can keep up with it, and help make it safe to refactor and upgrade.

BTW Verify and Testy can get on board if they A> import Test::Unit::Assertions (like certain other test rigs we could mention should), and B> implement flunk(). Both of those are all a custom assertion should ever need...

···

--
   Phlip
   http://www.zeroplayer.com/

20

Maybe, indeed I had the very strong intention to write Lab419 to make
my code Ruby1.9 and 1.8 compatible. Too much work? Not enough time? I
dunno, finally I got tired of it and wanted my closure based Verify to
work. This goes without pain in 1.9.
But I agree with what you say.
<snip>
R.

···

On Sat, Apr 4, 2009 at 1:14 AM, Tom Cloyd <tomcloyd@comcast.net> wrote:

Robert Dober wrote:

But this 1.9 nonsense frustrates me. Much as I want to dive in (it's
installed and ready to go), I simply don't have time to move code I depend
upon into an environment where I don't know whether it will fly or not, or
whether it will splat after 3 days of investment and use. I wish there was
somewhere I could go to find out what gems run with 1.9 and what do not.
But, that seems not to be the ruby way. That's plain crazy, say I.

--
There are some people who begin the Zoo at the beginning, called
WAYIN, and walk as quickly as they can past every cage until they get
to the one called WAYOUT, but the nicest people go straight to the
animal they love the most, and stay there. ~ A.A. Milne (from
Winnie-the-Pooh)