[ANN] rcov 0.7.0 (code coverage for Ruby)

ruby-talk
1

Source code, additional information, screenshots... available at
    http://eigenclass.org/hiki.rb?rcov
Release information:
    http://eigenclass.org/hiki.rb?rcov+0.7.0

Sample (fully) cross-referenced report at
    http://eigenclass.org/static/rcov-sample-report-full-crossref/

This release includes two RubyGems packages: a binary one for Win32 and a
platform-independent one for all those with a compiler, (or a lot of patience,
if willing to run rcov in pure-Ruby mode), so
  gem install rcov
should work once the packages have propagated to the mirror network (as of
14:20 UTC, about 5H since I uploaded them to Rubyforge, they haven't).

Overview

···

========
rcov is a code coverage tool for Ruby. It is commonly used for viewing overall
test coverage of target code. It features:
* fast execution: 20-300 times faster than previous tools
* multiple analysis modes: standard, bogo-profile, "intentional testing",
  dependency analysis...
* detection of uncovered code introduced since the last run ("differential
  code coverage")
* fairly accurate coverage information through code linkage inference using
  simple heuristics
* cross-referenced XHTML and several kinds of text reports
* support for easy automation with Rake and Rant
* colorblind-friendliness

What's new in 0.7.0

See http://eigenclass.org/hiki.rb?rcov+0.7.0 for the detailed change summary.

Features
--------
* coverage/callsite data from multiple runs can be aggregated (--aggregate)

Bugfixes
--------
* the SCRIPT_LINES__ workaround works better
* fixed silly bug in coverage data acquisition (line after the correct one
  marked in some situations)
* avoid problems with repeated path separators in default ignore list, based
  on rbconfig's data

How do I use it?

In the common scenario, your tests are under test/ and the target code
(whose coverage you want) is in lib/. In that case, all you have to do is
use rcov to run the tests (instead of testrb), and a number of XHTML files
with the code coverage information will be generated, e.g.

    rcov -Ilib test/*.rb

will execute all the .rb files under test/ and generate the code coverage
report for the target code (i.e. for the files in lib/) under coverage/. The
target code needs not be under lib/; rcov will detect is as long as it is
require()d by the tests. rcov is smart enough to ignore "uninteresting"
files: the tests themselves, files installed in Ruby's standard locations,
etc. See rcov --help for the list of regexps rcov matches filenames
against.

rcov can also be used from Rake; see README.rake or the RDoc documentation
for more information.

rcov can output information in several formats, and perform different kinds
of analyses in addition to plain code coverage. See rcov --help for a
description of the available options.

Sample output

See http://eigenclass.org/hiki.rb?rcov (once again) for screenshots.

The text report (also used by default in RcovTasks) resembles

+-----------------------------------------------------+-------+-------+--------+

                 File | Lines | LOC | COV |

+-----------------------------------------------------+-------+-------+--------+

lib/rcov.rb | 572 | 358 | 91.3% |

+-----------------------------------------------------+-------+-------+--------+

Total | 572 | 358 | 91.3% |

+-----------------------------------------------------+-------+-------+--------+
91.3% 1 file(s) 572 Lines 358 LOC

The (undecorated) textual output with execution count information looks like this:

$ rcov --no-html --text-counts b.rb

./b.rb

                                                                       > 2
a, b, c = (1..3).to_a | 2
10.times do | 1
   a += 1 | 10
   20.times do |i| | 10
     b += i | 200
     b.times do | 200
       c += (j = (b-a).abs) > 0 ? j : 0 | 738800
     end | 0
   end | 0
end | 0

rcov can detect when you've added code that was not covered by your unit
tests:

    $ rcov --text-coverage-diff --no-color test/*.rb
    Started
    .......................................
    Finished in 1.163085 seconds.
    
    39 tests, 415 assertions, 0 failures, 0 errors
    
    ================================================================================
    !!!!! Uncovered code introduced in lib/rcov.rb
    
    ### lib/rcov.rb:207
       
   def precompute_coverage(comments_run_by_default = true)
     changed = false
     lastidx = lines.size - 1
     if (!is_code?(lastidx) || /^__END__$/ =~ @lines[-1]) && !@coverage[lastidx]
    !! # mark the last block of comments
    !! @coverage[lastidx] ||= :inferred
    !! (lastidx-1).downto(0) do |i|
    !! break if is_code?(i)
    !! @coverage[i] ||= :inferred
    !! end
    !! end
     (0...lines.size).each do |i|
       next if @coverage[i]
       line = @lines[i]

Thanks

Thomas Leitner:
* reported that the SCRIPT_LINES__ workaround did not always work

Assaph Mehr:
* beta-tested 0.7.0 and found a bug in --aggregate (missing files)

Ryan Kinderman:
* suggested that -Ipath be passed to ruby instead of rcov in RcovTasks

License
-------
rcov is released under the terms of Ruby's license.
rcov includes xx 0.1.0, which is subject to the following conditions:

ePark Labs Public License version 1
Copyright (c) 2005, ePark Labs, Inc. and contributors
All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this
     list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright notice,
     this list of conditions and the following disclaimer in the documentation
     and/or other materials provided with the distribution.
  3. Neither the name of ePark Labs nor the names of its contributors may be
     used to endorse or promote products derived from this software without
     specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--
Mauricio Fernandez - http://eigenclass.org - singular Ruby

2

I am wondering, for a rails project, would this be the correct way to use
this?

rcov -Ilib test/**/*.rb

That works for me, albeit about 30% of my tests fail then, while when I run
them using the rake task ("rake"), they all pass. So I suppose I must be
doing something wrong.

Thanks alot, great project,

Rob

···

On 8/4/06, Mauricio Fernandez <mfp@acm.org> wrote:

Source code, additional information, screenshots... available at
    http://eigenclass.org/hiki.rb?rcov
Release information:
    http://eigenclass.org/hiki.rb?rcov+0.7.0

Sample (fully) cross-referenced report at
    http://eigenclass.org/static/rcov-sample-report-full-crossref/

This release includes two RubyGems packages: a binary one for Win32 and a
platform-independent one for all those with a compiler, (or a lot of
patience,
if willing to run rcov in pure-Ruby mode), so
  gem install rcov
should work once the packages have propagated to the mirror network (as of
14:20 UTC, about 5H since I uploaded them to Rubyforge, they haven't).

Overview

rcov is a code coverage tool for Ruby. It is commonly used for viewing
overall
test coverage of target code. It features:
* fast execution: 20-300 times faster than previous tools
* multiple analysis modes: standard, bogo-profile, "intentional testing",
  dependency analysis...
* detection of uncovered code introduced since the last run ("differential
  code coverage")
* fairly accurate coverage information through code linkage inference
using
  simple heuristics
* cross-referenced XHTML and several kinds of text reports
* support for easy automation with Rake and Rant
* colorblind-friendliness

What's new in 0.7.0

See http://eigenclass.org/hiki.rb?rcov+0.7.0 for the detailed change
summary.

Features
--------
* coverage/callsite data from multiple runs can be aggregated
(--aggregate)

Bugfixes
--------
* the SCRIPT_LINES__ workaround works better
* fixed silly bug in coverage data acquisition (line after the correct one
  marked in some situations)
* avoid problems with repeated path separators in default ignore list,
based
  on rbconfig's data

How do I use it?

In the common scenario, your tests are under test/ and the target code
(whose coverage you want) is in lib/. In that case, all you have to do is
use rcov to run the tests (instead of testrb), and a number of XHTML files
with the code coverage information will be generated, e.g.

    rcov -Ilib test/*.rb

will execute all the .rb files under test/ and generate the code coverage
report for the target code (i.e. for the files in lib/) under coverage/.
The
target code needs not be under lib/; rcov will detect is as long as it is
require()d by the tests. rcov is smart enough to ignore "uninteresting"
files: the tests themselves, files installed in Ruby's standard locations,
etc. See rcov --help for the list of regexps rcov matches filenames
against.

rcov can also be used from Rake; see README.rake or the RDoc documentation
for more information.

rcov can output information in several formats, and perform different
kinds
of analyses in addition to plain code coverage. See rcov --help for a
description of the available options.

Sample output

See http://eigenclass.org/hiki.rb?rcov (once again) for screenshots.

The text report (also used by default in RcovTasks) resembles

+-----------------------------------------------------+-------+-------+--------+
> File | Lines
> LOC | COV |

+-----------------------------------------------------+-------+-------+--------+
>lib/rcov.rb | 572 | 358 |
91.3% |

+-----------------------------------------------------+-------+-------+--------+
>Total | 572 | 358 |
91.3% |

+-----------------------------------------------------+-------+-------+--------+
91.3% 1 file(s) 572 Lines 358 LOC

The (undecorated) textual output with execution count information looks
like this:

$ rcov --no-html --text-counts b.rb

================================================================================
./b.rb

================================================================================

> 2
a, b, c = (1..3).to_a
> 2
10.times do
> 1
   a +=
1 | 10
   20.times do |i|
> 10
     b +=
i | 200
     b.timesdo | 200
       c += (j = (b-a).abs) > 0 ? j : 0 |
738800
     end
> 0
   end
> 0
end
> 0

rcov can detect when you've added code that was not covered by your unit
tests:

    $ rcov --text-coverage-diff --no-color test/*.rb
    Started
    .......................................
    Finished in 1.163085 seconds.

    39 tests, 415 assertions, 0 failures, 0 errors

    ================================================================================
    !!!!! Uncovered code introduced in lib/rcov.rb

    ### lib/rcov.rb:207

         def precompute_coverage(comments_run_by_default = true)
           changed = false
           lastidx = lines.size - 1
           if (!is_code?(lastidx) || /^__END__$/ =~ @lines[-1]) &&
!@coverage[lastidx]
    !! # mark the last block of comments
    !! @coverage[lastidx] ||= :inferred
    !! (lastidx-1).downto(0) do |i|
    !! break if is_code?(i)
    !! @coverage[i] ||= :inferred
    !! end
    !! end
           (0...lines.size).each do |i|
             next if @coverage[i]
             line = @lines[i]

Thanks

Thomas Leitner:
* reported that the SCRIPT_LINES__ workaround did not always work

Assaph Mehr:
* beta-tested 0.7.0 and found a bug in --aggregate (missing files)

Ryan Kinderman:
* suggested that -Ipath be passed to ruby instead of rcov in RcovTasks

License
-------
rcov is released under the terms of Ruby's license.
rcov includes xx 0.1.0, which is subject to the following conditions:

ePark Labs Public License version 1
Copyright (c) 2005, ePark Labs, Inc. and contributors
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification,
are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright
notice, this
     list of conditions and the following disclaimer.
  2. Redistributions in binary form must reproduce the above copyright
notice,
     this list of conditions and the following disclaimer in the
documentation
     and/or other materials provided with the distribution.
  3. Neither the name of ePark Labs nor the names of its contributors may
be
     used to endorse or promote products derived from this software
without
     specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED
WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
LIABLE FOR
ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES
(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
SERVICES;
LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON
ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
THIS
SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

--
Mauricio Fernandez - http://eigenclass.org - singular Ruby

3

http://www.agilewebdevelopment.com/plugins/rails_rcov

That plugin takes a lot of the headache out of using rails + rcov.
(Please don't top post).

···

On Aug 6, 2006, at 11:15 PM, Robert MannI wrote:

I am wondering, for a rails project, would this be the correct way to use
this?

rcov -Ilib test/**/*.rb

That works for me, albeit about 30% of my tests fail then, while when I run
them using the rake task ("rake"), they all pass. So I suppose I must be
doing something wrong.

Thanks alot, great project,

4

This was actually discovered on a rails project. In Rails case there
seems to be some weird interaction where running all tests
(integreation and functional for me) causes clashes. The rake tasks
for rails also launch a separate test process for each, and now rcov
mimics that.

Using the coverage.task template provided by Mauricio for rails seems
to fix the issue.

HTH,
Assaph

···

On 8/7/06, Robert MannI <robmnl@gmail.com> wrote:

I am wondering, for a rails project, would this be the correct way to use
this?

rcov -Ilib test/**/*.rb

That works for me, albeit about 30% of my tests fail then, while when I run
them using the rake task ("rake"), they all pass. So I suppose I must be
doing something wrong.

5

As Assaph said, this can fail in some circumstances, the reason being that
unit, functional and integration tests cannot always be run in the same
process safely; this is why the test task in Rails apps executes them
separately.

So, if you are lucky and there are no clashes, just

  rcov --rails -Ilib test/**/*_test.rb

would do. --rails tells rcov to ignore config/, environment/ and vendor/ in
the statistics.

Now, if you see that the outcome of the tests when run as shown above differs
from the results given by rake test (as happened to you), you have to execute
unit, functional and integration tests in three different processes.
You can do it manually as follows:

  rm -f coverage.data
  rcov --no-html --rails --aggregate coverage.data -Ilib test/unit/*_test.rb
  rcov --no-html --rails --aggregate coverage.data -Ilib test/functional/*_test.rb
  rcov --rails --aggregate coverage.data -Ilib test/integration/*_test.rb

--aggregate is used to merge the coverage data from the three runs in a
single report; --no-html is used in the first two runs because anyway the
last execution will create the report with all the information.

If you want a recipe to automate that, have a look at
http://eigenclass.org/hiki.rb?rcov+0.7.0
where I show a small snippet you can dump into e.g. lib/tasks/rcov.rake.
It will generate a number of tasks in the test:coverage namespace, as well as
a test:coverage task that runs all the tests under rcov.

Unfortunately, --aggregate is relatively slow because it has to save quite a
lot of information across runs: all the Ruby code that was parsed, the
coverage data, etc. There's no way around that though. Of course, you can also
create separate reports for unit, functional and integration tests (using -o
to choose the destination dirs for the reports, and omitting --aggregate),
but you won't be able to obtain a single unified coverage rate that way.

HTH,

···

On Mon, Aug 07, 2006 at 12:15:30PM +0900, Robert MannI wrote:

I am wondering, for a rails project, would this be the correct way to use
this?

rcov -Ilib test/**/*.rb

That works for me, albeit about 30% of my tests fail then, while when I run
them using the rake task ("rake"), they all pass. So I suppose I must be
doing something wrong.

--
Mauricio Fernandez - http://eigenclass.org - singular Ruby

6

Mauricio thanks for the excellent tool. I use it every day.

Michael

7

Mauricio, the coverage tasks is awesome.

It is working for me now.

Speed is not an issue for me, as I only do coverage runs once in a while.

Thank you,
Rob

···

On 8/7/06, Mauricio Fernandez <mfp@acm.org> wrote:

On Mon, Aug 07, 2006 at 12:15:30PM +0900, Robert MannI wrote:
> I am wondering, for a rails project, would this be the correct way to
use
> this?
>
> rcov -Ilib test/**/*.rb
>
> That works for me, albeit about 30% of my tests fail then, while when I
run
> them using the rake task ("rake"), they all pass. So I suppose I must be
> doing something wrong.

As Assaph said, this can fail in some circumstances, the reason being that
unit, functional and integration tests cannot always be run in the same
process safely; this is why the test task in Rails apps executes them
separately.

So, if you are lucky and there are no clashes, just

  rcov --rails -Ilib test/**/*_test.rb

would do. --rails tells rcov to ignore config/, environment/ and vendor/
in
the statistics.

Now, if you see that the outcome of the tests when run as shown above
differs
from the results given by rake test (as happened to you), you have to
execute
unit, functional and integration tests in three different processes.
You can do it manually as follows:

  rm -f coverage.data
  rcov --no-html --rails --aggregate coverage.data -Ilib
test/unit/*_test.rb
  rcov --no-html --rails --aggregate coverage.data -Ilib
test/functional/*_test.rb
  rcov --rails --aggregate coverage.data -Ilib test/integration/*_test.rb

--aggregate is used to merge the coverage data from the three runs in a
single report; --no-html is used in the first two runs because anyway the
last execution will create the report with all the information.

If you want a recipe to automate that, have a look at
http://eigenclass.org/hiki.rb?rcov+0.7.0
where I show a small snippet you can dump into e.g. lib/tasks/rcov.rake.
It will generate a number of tasks in the test:coverage namespace, as well
as
a test:coverage task that runs all the tests under rcov.

Unfortunately, --aggregate is relatively slow because it has to save quite
a
lot of information across runs: all the Ruby code that was parsed, the
coverage data, etc. There's no way around that though. Of course, you can
also
create separate reports for unit, functional and integration tests (using
-o
to choose the destination dirs for the reports, and omitting --aggregate),
but you won't be able to obtain a single unified coverage rate that way.

HTH,
--
Mauricio Fernandez - http://eigenclass.org - singular Ruby

8

Hi,

sorry for being -- may be -- a little offtopic.

Currently I am playing around with """""encryption""""" -- the
meaning of all those "'s is: "Encryption" in the sense of "convert
readable text into unreadable text" not in the sense of "Bullet
proof encryption suitable for secret service agents".

Just something to play around with...

What I need is the following "device":

Input is a freely defineable range of anything a byte can stand
for. Output is again a value of the same range. This mapping from an
input to an output value should be made variable in two manners:
First is to change the mapping itsself the second way is to define
an offset. This should work as follows:

Without offset: With offset 3:

A -> Z A -> I // wrapped around from the end
B -> H B -> B // of the previously defined
C -> L C -> Q // range
D -> P D -> Z
E -> F E -> H
F -> D F -> L
. G -> P
. H -> F
. I -> D

The offset should be changeable at each new mapping action.

My first idea was to define an Hash which maps an input range
to an index. To this index it is easy to add an offset. The
second step would be an array, which maps the new calculated index
(old index + offset) to the same range again, but this time
the range was "sorted" by sort{rand} st initialization time.

But when I do a "a=device.new", what should I return as an obejct
to the caller ??? TWO objects -- the hash AND the array ?

I am curious whether there are better, more elegant ways to accomplish
this in a more rubyish way.

I would be happy to read of your ideas how to implement this! :slight_smile:
Thank you very much in advance for any help and any idea !

Ruby! :slight_smile:
mcc

9

The whole point of wrapping something like this is in an object is that these are just internal details user code for the object never needs to worry about. For example, users are going to call your code something like:

   dev = Device.new( .. )
   after = dev.encryrpt(before)

All they know is that they are working with a Device. They don't need to worry about Hashes and Arrays.

I recommend using a Hash internally in the object. When a problem says "mapping" that usually means Hash. You could build the interface to handle either a Hash or an Integer being passed to the constructor:

   class Device
     def initialize(mapping_or_offset)
       if mapping_or_offset.is_a? Hash
         @mapping = build_mapping_from_hash(mapping_or_offset)
       else # offset
         @mapping = build_mapping_from_integer(mapping_or_offset)
       end
     end

     def encrypt(bytes)
       # use @mapping here...
     end

     private

     def build_mapping_from_hash(hash)
       # ...
     end

     def build_mapping_from_integer(integer)
       # ...
     end
   end

Hope that gives you some fresh ideas.

James Edward Gray II

···

On Aug 6, 2006, at 11:29 PM, Meino Christian Cramer wrote:

My first idea was to define an Hash which maps an input range
to an index. To this index it is easy to add an offset. The
second step would be an array, which maps the new calculated index
(old index + offset) to the same range again, but this time
the range was "sorted" by sort{rand} st initialization time.

But when I do a "a=device.new", what should I return as an obejct
to the caller ??? TWO objects -- the hash AND the array ?