[ANN] rcov 0.4.0 (code coverage) - scriptability, accuracy, unrotten internals

ruby-talk
1

Source code, additional information, screenshots... available at
    http://eigenclass.org/hiki.rb?rcov
Release information:
    http://eigenclass.org/hiki.rb?rcov+0.4.0

You can find a sample code coverage report generated by rcov at
    http://eigenclass.org/static/rcov-sample-report/

If you're on win32, you can also find a pre-built rcovrt.so (which makes
code coverage analysis >100 times faster) in the above-mentioned pages.

Overview

···

========
rcov is a tool for simple code coverage analysis in Ruby. It features:
* fast execution: 20-300 times faster than previous tools
* multiple analysis modes
* fairly accurate coverage information through code linkage inference using
  simple heuristics
* XHTML and several kinds of text reports
* easy automation with Rake via a RcovTask
* colorblind-friendliness

What's new in 0.4.0

See http://eigenclass.org/hiki.rb?rcov+0.4.0 for the detailed change summary.

rcov 0.4.0 is really feature-packed, and (hopefully) less buggy than previous
releases. This can be ascribed in large part to the use of rcov on itself to
achieve better test coverage. Nearly 100 changesets went into rcov 0.4.0.

Starting with the most obvious changes, the XHTML output looks much better
now with the new color-scheme. Plain text output formats have also improved
considerably.

rcov 0.4.0 is much easier to use from external scripts, and invoking rcov
from your Rakefile takes no effort with the included Rcov::RcovTask; it can
take as little as:

    require 'rcov/rcovtask'
    Rcov::RcovTask.new

Really! This creates a task named "rcov" which will run the tests matching
test/test*.rb and create a XHTML report. And a nice textual report on
stdout). And a clobber_rcov task, which the main clobber target will depend
on, to delete the former.

However, sometimes that lone Rcov::RcovTask.new will not suffice; this would
be a more realistic example:

    # if you don't like the default description
    desc "Analyze code coverage of the unit tests."
    Rcov::RcovTask.new do |t|
      # change the glob pattern in the next line
      t.test_files = FileList['test/test*.rb']
      t.rcov_opts << "--sort coverage" # sort by increasing coverage rate
    end

See README.rake in the distribution tarball (or the RDoc documentation, which
includes it) for more information about Rcov::RcovTask.

Parts of the code have been moved into a reusable library that allows you to
build on top of rcov's coverage analysis capabilities. This change allowed for
much better testing of rcov's internals, which helped squash many bugs and add
features while making the overall code look better thanks to some long-needed
refactoring.

Short summary of user-visible changes
--------------------------------------

* easy automation via Rake using the bundled Rcov::RcovTask
* better heuristics: supports heredocs at last, including variants your editor
  is probably unable to handle, and =begin/=end comments
* --rails option to ignore files under vendor/, enviroment/ and config/
* parts of the runtime exposed and documented for external use
* new color scheme, with alternating shades of the base colors
* -w to set $VERBOSE=true (turns on warnings)
* --text-report and --text-summary
* --sort and --sort-reverse for the summaries and reports
* --threshold and --only-uncovered
* --replace-progname

Backwards incompatible changes
------------------------------
* renamed --text to --text-counts and --rich-text to --text-coverage: they
  were misnamed to begin with
* changed the meaning of -t and -T (--text-summary and --text-report)
* $0 is not changed by default for each loaded file anymore; the old
  behavior (modulo a small bugfix) can be reproduced with --replace-progname

Thanks (do tell if I forgot to list your name)

Robert Feldt:
* helped to refine the color scheme
* very valuable suggestions; a large percentage of the new functionality in
  0.4.0 is a consequence of his requests

Andre Nathan:
* identified a couple bugs in the heuristics that eluded my testing

Downloading

The last version is available at
  http://eigenclass.org/hiki.rb?rcov

How do I use it?

Just use rcov to run your program (instead of ruby), and a number of XHTML
files with the code coverage information will be generated, e.g.

rcov test/*.rb

will execute all the .rb files under test/ and generate the code coverage report
under coverage/. rcov can also operate in "bogo-profiling mode"
and output the relevant information in alternative formats. Use
rcov -h
for more information.

Sample output

See http://eigenclass.org/hiki.rb?rcov (once again) for screenshots.
A sample code coverage report generated by rcov is available at
    http://eigenclass.org/static/rcov-sample-report/

Text reports (also used by default in RcovTasks) resemble

+-----------------------------------------------------+-------+-------+--------+

                 File | Lines | LOC | COV |

+-----------------------------------------------------+-------+-------+--------+

app/controllers/application.rb | 39 | 29 | 31.0% |
app/helpers/application_helper.rb | 147 | 119 | 23.5% |
app/models/aggregations/tada.rb | 75 | 45 | 31.1% |
app/models/aggregations/upcoming.rb | 78 | 48 | 29.2% |
app/models/article.rb | 109 | 78 | 67.9% |
app/models/category.rb | 30 | 23 | 60.9% |
app/models/sidebar.rb | 36 | 27 | 40.7% |
components/plugins/sidebars/archives_controller.rb | 35 | 27 | 29.6% |
components/plugins/sidebars/category_controller.rb | 20 | 16 | 50.0% |
components/plugins/sidebars/delicious_controller.rb | 20 | 16 | 50.0% |
components/plugins/sidebars/flickr_controller.rb | 20 | 16 | 50.0% |
components/plugins/sidebars/fortythree_controller.rb | 20 | 16 | 50.0% |
...s/plugins/sidebars/fortythreeplaces_controller.rb | 20 | 16 | 50.0% |
components/plugins/sidebars/static_controller.rb | 27 | 24 | 29.2% |
components/plugins/sidebars/tada_controller.rb | 20 | 16 | 50.0% |
components/plugins/sidebars/technorati_controller.rb | 20 | 16 | 50.0% |
components/plugins/sidebars/upcoming_controller.rb | 20 | 16 | 50.0% |
components/plugins/sidebars/xml_controller.rb | 16 | 13 | 53.8% |
components/sidebars/sidebar_controller.rb | 110 | 80 | 32.5% |
lib/html_engine.rb | 29 | 23 | 78.3% |
lib/login_system.rb | 85 | 34 | 23.5% |
lib/migrator.rb | 28 | 22 | 40.9% |
lib/renderfix.rb | 32 | 25 | 16.0% |
lib/xmlrpc_fix.rb | 13 | 12 | 25.0% |

+-----------------------------------------------------+-------+-------+--------+

Total | 1754 | 1251 | 60.1% |

+-----------------------------------------------------+-------+-------+--------+
60.1% 42 file(s) 1754 Lines 1251 LOC

This was run with --threshold 80, meaning that only files under 80% coverage
are shown in the table, but the overall statistics include all analyzed files.

The textual output with execution count information looks like this:

$ rcov --no-html --text-counts b.rb

./b.rb

                                                                       > 2
a, b, c = (1..3).to_a | 2
10.times do | 1
   a += 1 | 10
   20.times do |i| | 10
     b += i | 200
     b.times do | 200
       c += (j = (b-a).abs) > 0 ? j : 0 | 738800
     end | 0
   end | 0
end | 0

License
-------
rcov is released under the terms of Ruby's license.
rcov includes xx 0.1.0, which is subject to the following conditions:

ePark Labs Public License version 1
Copyright (c) 2005, ePark Labs, Inc. and contributors
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this
     list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice,
     this list of conditions and the following disclaimer in the documentation
     and/or other materials provided with the distribution.
  3. Neither the name of ePark Labs nor the names of its contributors may be
     used to endorse or promote products derived from this software without
     specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--
Mauricio Fernandez - http://eigenclass.org - singular Ruby

2

This is awesome. It's enlightening and sad to see the coverage of some of my unit tests, but without a doubt I'm hooked. Visually looking at the test coverage (or lack thereof) is really much more effective than I thought. If anything this will help to be a lot more conscious about creating solid test suites, rather than cherry picking the easy/obvious ones that most likely don't do the job like they should.

When do we get a gem?

Thanks a lot,
Jeff

Mauricio Fernandez wrote:

···

Source code, additional information, screenshots... available at http://eigenclass.org/hiki.rb?rcov
Release information:
    http://eigenclass.org/hiki.rb?rcov+0.4.0

You can find a sample code coverage report generated by rcov at
    http://eigenclass.org/static/rcov-sample-report/

If you're on win32, you can also find a pre-built rcovrt.so (which makes
code coverage analysis >100 times faster) in the above-mentioned pages.

Overview

rcov is a tool for simple code coverage analysis in Ruby. It features:
* fast execution: 20-300 times faster than previous tools
* multiple analysis modes
* fairly accurate coverage information through code linkage inference using simple heuristics
* XHTML and several kinds of text reports
* easy automation with Rake via a RcovTask * colorblind-friendliness

What's new in 0.4.0

See http://eigenclass.org/hiki.rb?rcov+0.4.0 for the detailed change summary.

rcov 0.4.0 is really feature-packed, and (hopefully) less buggy than previous
releases. This can be ascribed in large part to the use of rcov on itself to achieve better test coverage. Nearly 100 changesets went into rcov 0.4.0.

Starting with the most obvious changes, the XHTML output looks much better
now with the new color-scheme. Plain text output formats have also improved
considerably.

rcov 0.4.0 is much easier to use from external scripts, and invoking rcov
from your Rakefile takes no effort with the included Rcov::RcovTask; it can
take as little as:

    require 'rcov/rcovtask'
    Rcov::RcovTask.new

Really! This creates a task named "rcov" which will run the tests matching
test/test*.rb and create a XHTML report. And a nice textual report on
stdout). And a clobber_rcov task, which the main clobber target will depend
on, to delete the former.

However, sometimes that lone Rcov::RcovTask.new will not suffice; this would
be a more realistic example:

    # if you don't like the default description
    desc "Analyze code coverage of the unit tests." Rcov::RcovTask.new do |t|
      # change the glob pattern in the next line
      t.test_files = FileList['test/test*.rb'] t.rcov_opts << "--sort coverage" # sort by increasing coverage rate
    end

See README.rake in the distribution tarball (or the RDoc documentation, which
includes it) for more information about Rcov::RcovTask.

Parts of the code have been moved into a reusable library that allows you to
build on top of rcov's coverage analysis capabilities. This change allowed for
much better testing of rcov's internals, which helped squash many bugs and add
features while making the overall code look better thanks to some long-needed
refactoring.

Short summary of user-visible changes
--------------------------------------

* easy automation via Rake using the bundled Rcov::RcovTask
* better heuristics: supports heredocs at last, including variants your editor
  is probably unable to handle, and =begin/=end comments
* --rails option to ignore files under vendor/, enviroment/ and config/
* parts of the runtime exposed and documented for external use
* new color scheme, with alternating shades of the base colors
* -w to set $VERBOSE=true (turns on warnings)
* --text-report and --text-summary
* --sort and --sort-reverse for the summaries and reports
* --threshold and --only-uncovered
* --replace-progname

Backwards incompatible changes
------------------------------
* renamed --text to --text-counts and --rich-text to --text-coverage: they
  were misnamed to begin with
* changed the meaning of -t and -T (--text-summary and --text-report)
* $0 is not changed by default for each loaded file anymore; the old behavior (modulo a small bugfix) can be reproduced with --replace-progname

Thanks (do tell if I forgot to list your name)

Robert Feldt:
* helped to refine the color scheme
* very valuable suggestions; a large percentage of the new functionality in
  0.4.0 is a consequence of his requests

Andre Nathan:
* identified a couple bugs in the heuristics that eluded my testing

Downloading

The last version is available at http://eigenclass.org/hiki.rb?rcov

How do I use it?

Just use rcov to run your program (instead of ruby), and a number of XHTML
files with the code coverage information will be generated, e.g.

rcov test/*.rb

will execute all the .rb files under test/ and generate the code coverage report
under coverage/. rcov can also operate in "bogo-profiling mode"
and output the relevant information in alternative formats. Use
rcov -h
for more information.

Sample output

See http://eigenclass.org/hiki.rb?rcov (once again) for screenshots.
A sample code coverage report generated by rcov is available at
    http://eigenclass.org/static/rcov-sample-report/

Text reports (also used by default in RcovTasks) resemble

+-----------------------------------------------------+-------+-------+--------+
> File | Lines | LOC | COV |
+-----------------------------------------------------+-------+-------+--------+
>app/controllers/application.rb | 39 | 29 | 31.0% |
>app/helpers/application_helper.rb | 147 | 119 | 23.5% |
>app/models/aggregations/tada.rb | 75 | 45 | 31.1% |
>app/models/aggregations/upcoming.rb | 78 | 48 | 29.2% |
>app/models/article.rb | 109 | 78 | 67.9% |
>app/models/category.rb | 30 | 23 | 60.9% |
>app/models/sidebar.rb | 36 | 27 | 40.7% |
>components/plugins/sidebars/archives_controller.rb | 35 | 27 | 29.6% |
>components/plugins/sidebars/category_controller.rb | 20 | 16 | 50.0% |
>components/plugins/sidebars/delicious_controller.rb | 20 | 16 | 50.0% |
>components/plugins/sidebars/flickr_controller.rb | 20 | 16 | 50.0% |
>components/plugins/sidebars/fortythree_controller.rb | 20 | 16 | 50.0% |
>...s/plugins/sidebars/fortythreeplaces_controller.rb | 20 | 16 | 50.0% |
>components/plugins/sidebars/static_controller.rb | 27 | 24 | 29.2% |
>components/plugins/sidebars/tada_controller.rb | 20 | 16 | 50.0% |
>components/plugins/sidebars/technorati_controller.rb | 20 | 16 | 50.0% |
>components/plugins/sidebars/upcoming_controller.rb | 20 | 16 | 50.0% |
>components/plugins/sidebars/xml_controller.rb | 16 | 13 | 53.8% |
>components/sidebars/sidebar_controller.rb | 110 | 80 | 32.5% |
>lib/html_engine.rb | 29 | 23 | 78.3% |
>lib/login_system.rb | 85 | 34 | 23.5% |
>lib/migrator.rb | 28 | 22 | 40.9% |
>lib/renderfix.rb | 32 | 25 | 16.0% |
>lib/xmlrpc_fix.rb | 13 | 12 | 25.0% |
+-----------------------------------------------------+-------+-------+--------+
>Total | 1754 | 1251 | 60.1% |
+-----------------------------------------------------+-------+-------+--------+
60.1% 42 file(s) 1754 Lines 1251 LOC

This was run with --threshold 80, meaning that only files under 80% coverage
are shown in the table, but the overall statistics include all analyzed files.

The textual output with execution count information looks like this:

$ rcov --no-html --text-counts b.rb ================================================================================
./b.rb

                                                                       > 2
a, b, c = (1..3).to_a | 2
10.times do | 1
   a += 1 | 10
   20.times do |i| | 10
     b += i | 200
     b.times do | 200
       c += (j = (b-a).abs) > 0 ? j : 0 | 738800
     end | 0
   end | 0
end | 0

License
-------
rcov is released under the terms of Ruby's license.
rcov includes xx 0.1.0, which is subject to the following conditions:

ePark Labs Public License version 1
Copyright (c) 2005, ePark Labs, Inc. and contributors
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this
     list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice,
     this list of conditions and the following disclaimer in the documentation
     and/or other materials provided with the distribution.
  3. Neither the name of ePark Labs nor the names of its contributors may be
     used to endorse or promote products derived from this software without
     specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

3

Mauricio, I'll hook you up with gem builds if you want. I use rcov a
lot and having gems makes it easier for me to get other developers to
use it.

···

On Tue, 2006-05-23 at 01:09 +0900, Jeff Rose wrote:

This is awesome. It's enlightening and sad to see the coverage of some
of my unit tests, but without a doubt I'm hooked. Visually looking at
the test coverage (or lack thereof) is really much more effective than I
thought. If anything this will help to be a lot more conscious about
creating solid test suites, rather than cherry picking the easy/obvious
ones that most likely don't do the job like they should.

When do we get a gem?

--
Zed A. Shaw

http://mongrel.rubyforge.org/

4

The one thing that has been blocking (or at least pushing down the TODO list)
the RubyGems package was how to combine it with cross-compilation for win32,
since I figured a RubyGems package would be of most value to win32 users who
have no access to a C toolchain. There are a few other problems I'd have liked
to solve, but they're much more general and probably addressable at RubyGems'
level (C runtime compatibility, ABI checking...).

A "normal" RubyGems package (i.e. one that requires a C compiler) should be
very easy, so I should be able to get one out fairly quickly.

···

On Tue, May 23, 2006 at 05:17:47AM +0900, Zed Shaw wrote:

On Tue, 2006-05-23 at 01:09 +0900, Jeff Rose wrote:
> This is awesome. It's enlightening and sad to see the coverage of some
> of my unit tests, but without a doubt I'm hooked. Visually looking at
> the test coverage (or lack thereof) is really much more effective than I
> thought. If anything this will help to be a lot more conscious about
> creating solid test suites, rather than cherry picking the easy/obvious
> ones that most likely don't do the job like they should.
>
> When do we get a gem?

Mauricio, I'll hook you up with gem builds if you want. I use rcov a
lot and having gems makes it easier for me to get other developers to
use it.

--
Mauricio Fernandez - http://eigenclass.org - singular Ruby

5

I was recently faced with the issue of packaging a library that needed
to execute cross platform but for which a windows binary was required
and wondered why, when a win32 library is required, we don't just
package it in a win32 directory inside the gem and require it when the
platform is win32. These binaries take up very little disk space
(which is cheap anyway) and I'd rather have a little unused windows
dll on my Linux systems than to bother with installing (and
maintaining) different gems for different platforms.

Okay, Jim, tell me I am crazy. (Or ignorant--that would be much worse.)

···

--
John-Mason Shackelford

Software Developer
Pearson Educational Measurement

2510 North Dodge St.
Iowa City, IA 52245
ph. 319-354-9200x6214
john-mason.shackelford@pearson.com
http://pearsonedmeasurement.com

--
John-Mason Shackelford

Software Developer
Pearson Educational Measurement

2510 North Dodge St.
Iowa City, IA 52245
ph. 319-354-9200x6214
john-mason.shackelford@pearson.com
http://pearsonedmeasurement.com

6

I got to generate the platform-independent package and the cross-compiled
binary one as follows:

PKG_REVISION = ".1"
PKG_FILES = FileList[
# sources, docs and Rakefile
# this is a modified rbconfig.rb based on that generated by a cross-compiled
# ruby targetting mingw32
"mingw-rbconfig.rb"
]

require 'rake/gempackagetask'
Spec = Gem::Specification.new do |s|
  # ...
  s.files = PKG_FILES.to_a
  s.extensions << "ext/rcovrt/extconf.rb"
  # ...
end

task :gem => [:test]
Rake::GemPackageTask.new(Spec) do |p|
  p.need_tar = true
end

# {{{ Cross-compilation and building of a binary RubyGems package for mswin32

require 'rake/clean'

WIN32_PKG_DIR = "rcov-" + Rcov::VERSION + PKG_REVISION

file "#{WIN32_PKG_DIR}" => [:package] do
  sh "tar zxf pkg/#{WIN32_PKG_DIR}.tgz"
end

# this overrides the normal rbconfig.rb, in order to generate a Makefile
# suitable for cross-compilation
desc "Cross-compile the rcovrt.so extension for win32"
file "rcovrt_win32" => ["#{WIN32_PKG_DIR}"] do
  cp "mingw-rbconfig.rb", "#{WIN32_PKG_DIR}/ext/rcovrt/rbconfig.rb"
  sh "cd #{WIN32_PKG_DIR}/ext/rcovrt/ && ruby -I. extconf.rb && make"
  mv "#{WIN32_PKG_DIR}/ext/rcovrt/rcovrt.so", "#{WIN32_PKG_DIR}/lib"
end

# New spec, essentially like the first one, modulo:
# * the platform, set to WIN32,
# * the removal of the extensions attribute, so that no compilation is
# attempted at install time
# * the addition of the desired .so to the list of files to be included

Win32Spec = Spec.clone
Win32Spec.platform = Gem::Platform::WIN32
Win32Spec.extensions =
Win32Spec.files += ["lib/rcovrt.so"]

desc "Build the binary RubyGems package for win32"
task :rubygems_win32 => ["rcovrt_win32"] do
  Dir.chdir("#{WIN32_PKG_DIR}") do
    Gem::Builder.new(Win32Spec).build
    verbose(true) {
      mv Dir["*.gem"].first, "../pkg/rcov-#{Rcov::VERSION + PKG_REVISION}-mswin32.gem"
    }
  end
end

CLEAN.include "#{WIN32_PKG_DIR}"

···

On Tue, May 23, 2006 at 07:32:41AM +0900, John-Mason P. Shackelford wrote:

I was recently faced with the issue of packaging a library that needed
to execute cross platform but for which a windows binary was required
and wondered why, when a win32 library is required, we don't just
package it in a win32 directory inside the gem and require it when the
platform is win32. These binaries take up very little disk space
(which is cheap anyway) and I'd rather have a little unused windows
dll on my Linux systems than to bother with installing (and
maintaining) different gems for different platforms.

Okay, Jim, tell me I am crazy. (Or ignorant--that would be much worse.)

--
Mauricio Fernandez - http://eigenclass.org - singular Ruby