Gems undermine unit testing

ruby-talk
1

I put the following line at the top of each of my unit tests for Watir:

   $LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..') if $0 == __FILE__

This ensures that the version of Watir that is being tested is the one that is in the same hierarchy as the unit tests (by putting its directory in the front of the load-path). You can accomplish the same thing by setting the load path at the command line.

... Unless you are using a gem. I recently started packaging Watir as a gem and i now see that, somehow, the gem mechanism prepends the gem library directories to the very front, so the gem-installed version is tested instead of the one under development.

Now i need to make sure that i do a "gem uninstall watir" before running unit tests or else they are going to be testing the wrong version -- the one that has already been tested and released. This is a type II error -- the tests say they've passed but in fact they haven't actually tested what they were supposed to and could easily miss errors.

Since i have many contributors to the project, i would like a more reliable mechanism to help ensure that the people running unit tests are actually testing the version under development.

Any suggestions?

Bret

···

_____________________
  Bret Pettichord
  www.pettichord.com

2

I ran into a similar situation recently and the workaround that I use is
to run a small batch file
before running all unit tests. The batch file has just two lines in it:

call gem build wet-web.gemspec
call gem install wet-web*.gem

pause

This ensures that the installed machine on my test machine is the
development version and not the
release version.

Raghu

Bret Pettichord wrote:

···

I put the following line at the top of each of my unit tests for Watir:

$LOAD_PATH.unshift File.join(File.dirname(FILE), ‘…’) if $0 == FILE

This ensures that the version of Watir that is being tested is the one that
is in the same hierarchy as the unit tests (by putting its directory in the
front of the load-path). You can accomplish the same thing by setting the
load path at the command line.

… Unless you are using a gem. I recently started packaging Watir as a gem
and i now see that, somehow, the gem mechanism prepends the gem library
directories to the very front, so the gem-installed version is tested
instead of the one under development.

Now i need to make sure that i do a “gem uninstall watir” before running
unit tests or else they are going to be testing the wrong version – the
one that has already been tested and released. This is a type II error –
the tests say they’ve passed but in fact they haven’t actually tested what
they were supposed to and could easily miss errors.

Since i have many contributors to the project, i would like a more reliable
mechanism to help ensure that the people running unit tests are actually
testing the version under development.

Any suggestions?

Bret

Bret Pettichord
www.pettichord.com

Wtr-general mailing list
Wtr-general-GrnCvJ7WPxnNLxjTenLetw@public.gmane.org
http://rubyforge.org/mailman/listinfo/wtr-general

–
Qantom Software

http://www.qantom.com
Ph : 91-80-26799269 Xtn. 125
sip : raghu-fYsKVbecyMPQT0dZR+AlfA@public.gmane.org

3

Bret,

I release all my software as gems They are installed on my system as gems and
yet am able to test all my software without contortions. I use the Rake test
task to run my tests (and it sets up the library paths on the command line).

If your software is available in the load path, gems will not attempt load the
gem in front of that. An explicit require_gem will try to load a particular
version of your library, so you don't want to use require_gem in your test
setup (and probably don't need it in the real code either).

···

On Tuesday 27 September 2005 10:12 pm, Bret Pettichord wrote:

I put the following line at the top of each of my unit tests for Watir:

   $LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..') if $0 ==
__FILE__

This ensures that the version of Watir that is being tested is the one that
is in the same hierarchy as the unit tests (by putting its directory in the
front of the load-path). You can accomplish the same thing by setting the
load path at the command line.

... Unless you are using a gem. I recently started packaging Watir as a gem
and i now see that, somehow, the gem mechanism prepends the gem library
directories to the very front, so the gem-installed version is tested
instead of the one under development.

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

4

The current version of RubyGems library (0.8.10) doesn't alter the LOADPATH. Instead it overrides Kernel#require with something like this:

module Kernel
  alias require__ require

  def require(path)
    require__ path
  rescue LoadError => load_error
    ... tries to require a gem ...
  end
end

So, it will always try to get stuff from LOADPATH first, only then go to RubyGems.

Earlier versions used to change LOADPATH, indeed. So, the answer to your question _may_ be as simple as "upgrade RubyGems"

Alex

5

I release all my software as gems They are installed on my system as gems and
yet am able to test all my software without contortions. I use the Rake test
task to run my tests (and it sets up the library paths on the command line).

I'll take a look at this. I've just started to use Rake.

If your software is available in the load path, gems will not attempt load the
gem in front of that. An explicit require_gem will try to load a particular
version of your library, so you don't want to use require_gem in your test
setup (and probably don't need it in the real code either).

I checked the load path in my testsuite, after executing "require 'watir'". And i saw that the gem path to watir was on the load path before the directory that had the development version. I don't use require_gem. But you are telling me that that wasn't supposed to happen.

After i uninstalled the watir gem, i was able to run my tests correctly.

Bret

···

At 10:26 PM 9/27/2005, Jim Weirich wrote:

_____________________
  Bret Pettichord
  www.pettichord.com

6

I, too, release all my software as gems. I don't, however, use the
Rake test task, finding that it works poorly on Windows. However, all
of my Rake files *do* run my tests inline. Look at what I do with
Text::Format for an example. The PDF::Writer Rake task is up to date
for this, I just don't yet have any unit tests for it :wink:

You *will* need to make sure that, for local installations, you put
your libraries into the $LOAD_PATH, otherwise RubyGems will find them.

-austin

···

On 9/27/05, Jim Weirich <jim@weirichhouse.org> wrote:

I release all my software as gems They are installed on my system as gems and
yet am able to test all my software without contortions. I use the Rake test
task to run my tests (and it sets up the library paths on the command line).

--
Austin Ziegler * halostatue@gmail.com
               * Alternate: austin@halostatue.ca

7

Alexey Verkhovsky wrote:

The current version of RubyGems library (0.8.10) doesn't alter the
LOADPATH. Instead it overrides Kernel#require with something like this:

module Kernel
  alias require__ require

  def require(path)
    require__ path
  rescue LoadError => load_error
    ... tries to require a gem ...
  end
end

So, it will always try to get stuff from LOADPATH first, only then go to
RubyGems.

This is true, but as Austin pointed out to me the other day, if there
is a match in Gems it will alter the LOAD_PATH so as not to have to
search again.

T.

8

Bret Pettichord wrote:

I checked the load path in my testsuite, after executing "require 'watir'". And i saw that the gem path to watir was on the load path before the directory that had the development version. I don't use require_gem. But you are telling me that that wasn't supposed to happen.

Hmmm... I don't know a lot about gems, but it's certainly counter to my setup:

C:\prog\rubycode>notepad binding_of_caller.rb

C:\prog\rubycode>type binding_of_caller.rb
puts <<'YAY!!!'
I'm a little teapot
Short and stout.
This is my handle.
This is my spout.

When I get all steamed up,
Hear me shout:
"Tip me over and
Pour me out!"
YAY!!!

C:\prog\rubycode>notepad test.rb

C:\prog\rubycode>type test.rb
require 'pp'
require 'binding_of_caller'
puts
pp $:

C:\prog\rubycode>ruby test.rb
I'm a little teapot
Short and stout.
This is my handle.
This is my spout.

When I get all steamed up,
Hear me shout:
"Tip me over and
Pour me out!"

["c:/prog/ruby/lib/ruby/site_ruby/1.8",
"c:/prog/ruby/lib/ruby/site_ruby/1.8/i386-msvcrt",
"c:/prog/ruby/lib/ruby/site_ruby",
"c:/prog/ruby/lib/ruby/1.8",
"c:/prog/ruby/lib/ruby/1.8/i386-mswin32",
"."]

C:\prog\rubycode>del binding_of_caller.rb

C:\prog\rubycode>ruby test.rb

["c:/prog/ruby/lib/ruby/gems/1.8/gems/rails-0.13.1/bin",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/rails-0.13.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionwebservice-0.8.1/lib/action_web_service/vendor/",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionwebservice-0.8.1/bin",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionwebservice-0.8.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionmailer-1.0.1/lib/action_mailer/vendor/",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionmailer-1.0.1/bin",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionmailer-1.0.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_view/helpers/../../action_controller/vendor/html-scanner",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/action_view/vendor",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib/../../activesupport/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionpack-1.9.1/bin",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/actionpack-1.9.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/activesupport-1.1.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/activerecord-1.11.1/lib/../../activesupport/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/activerecord-1.11.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/activerecord-1.11.1/bin",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/activerecord-1.11.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/activesupport-1.1.1/bin",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/activesupport-1.1.1/lib",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/rake-0.5.4/bin",
"c:/prog/ruby/lib/ruby/gems/1.8/gems/rake-0.5.4/lib",
"c:/prog/ruby/lib/ruby/site_ruby/1.8",
"c:/prog/ruby/lib/ruby/site_ruby/1.8/i386-msvcrt",
"c:/prog/ruby/lib/ruby/site_ruby",
"c:/prog/ruby/lib/ruby/1.8",
"c:/prog/ruby/lib/ruby/1.8/i386-mswin32",
"."]

After i uninstalled the watir gem, i was able to run my tests correctly.

BTW, using gems doesn't preclude being version-specific, unless you are using Edge Watir a lot, and dealing with cases that aren't packaged as gems:

http://docs.rubygems.org/read/chapter/4#page71

Devin

9

I'll take a look at this. I've just started to use Rake.

Let me know if you need any help with it.

I checked the load path in my testsuite, after executing "require 'watir'".
And i saw that the gem path to watir was on the load path before the
directory that had the development version. I don't use require_gem. But
you are telling me that that wasn't supposed to happen.

After i uninstalled the watir gem, i was able to run my tests correctly.

Then I am mystified. *Something* is causing the gem to be loaded. The fact
that the tests run correctly without the gem installed indicate the it is not
activated by need (since obviously you don't need it), nor by require_gem.

Does the watir zip file contain all the tests to reproduce this behavior?
Perhaps I'll take a look today.

···

On Wednesday 28 September 2005 04:07 am, Bret Pettichord wrote:

--
-- Jim Weirich jim@weirichhouse.org http://onestepback.org
-----------------------------------------------------------------
"Beware of bugs in the above code; I have only proved it correct,
not tried it." -- Donald Knuth (in a memo to Peter van Emde Boas)

10

I was using breakpoint to try and debug this situation, but i now think that it may have had side effects itself that confused me more.

I did
   require 'breakpoint'; breakpoint

In my script, and then from irb, checked $LOAD_PATH. That was when i saw that the directory to the watir gem library preceeded the 'current directory' that i had prepended to it.

I'm now wondering whether i in fact was getting the right watir library through require because the load path didn't get changed until later. But if this is true, then if i had required a gem library before the watir require, i would not have loaded the one i wanted.

It seems that the gem mechanism changes the load path to speed up further requires. But i am wondering if it modifies it more drastically than it needs to. If i am require'ing 'breakpoint', the paths to other gem libraries should not be showing up on the load path.

This is all somewhat speculative, because while i was trying to track this down, i started getting a different version of breakpoint than i intended. There apparently is a 'breakpoint' library in one of the rails gems and it was being loaded instead and it was redefining the module class in a way that made the modules in my code break. I ended up uninstalling a dozen gems that i had installed last week because i knew i was running into interaction problems and i needed to simplify my situation.

Bret

···

At 06:13 AM 9/28/2005, Jim Weirich wrote:

Then I am mystified. *Something* is causing the gem to be loaded. The fact
that the tests run correctly without the gem installed indicate the it is not
activated by need (since obviously you don't need it), nor by require_gem.

_____________________
  Bret Pettichord
  www.pettichord.com