Ruby Coverage Validator, a coverage tool for Ruby.
This tool is now in beta.
http://www.softwareverify.com/rubyCoverageValidator/index.html
Stephen
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk
RSI Information: http://www.objmedia.demon.co.uk/rsi.html
Stephen Kellett wrote:
Ruby Coverage Validator, a coverage tool for Ruby.
How does this compare to free solutions like rcov? I especially wonder if you are using trace_funcs (which can be quite slow) or if you found another way of getting the statistics. (The feature list states that it "Uses the tracing and profiling API introduced with Ruby 2.2." -- sounds like Python was replaced with Ruby and the version number was left untouched. :))
How does this compare to free solutions like rcov?
I cannot tell you, I have not used rcov. I just tried Google for "rcov Ruby Coverage" and couldn't find anything that I thought was the right target. If you are referring to a tool that produces text output and doesn't give you the output until the tool has ended there is no comparison - both give you coverage results, but one gives you a lot more insight into what is happening and how to represent the results and/or direct your testing, plus providing support for regression testing.
Ruby Coverage Validator graphical, stats updated in real time as the app executes - enabling you to direct your testing sequence to ensure maximum coverage if you are running interactively rather than as a regression test. You can run merge results from one run into another, or a central session - ideal for regression testing. You can export the results in HTML or XML. Multiple views onto the same data, etc.
I especially wonder if you are using trace_funcs (which can be quite slow)
Yes. Although the slowness would be compounded by using a trace func written in Ruby. Our trace func is written in C++. Matz and a few helpful people in this newsgroup provided enough information for us to put things together after examining the source code and writing quite a few test applications.
or if you found another way of getting the statistics.
Examining the Ruby source and the exported functions from the Ruby DLL indicate that it is not easy to access certain data structures that are not exported - easier and more future proof to use the existing API.
(The feature list states that it "Uses the tracing and profiling API introduced with Ruby 2.2." -- sounds like Python was replaced with Ruby and the version number was left untouched. :))
An unfortunate error. The user interface for the Python and Ruby software tools are very similar (as are the Java and C++ coverage software tools), hence the descriptions are similar - so the web descriptions are similar as are the images - check them, they are not the same - the Python images are for Python programs and the Ruby images are for Ruby programs.
We wrote the Python version 9 months ago. It will be released it soon, although that is down to Software Verification.
Aside: We are not interested in the "Is Python better/worse than Ruby/Java/whatever" debate. It doesn't get anyone anywhere, except distracted from using their own preference in language choice.
Stephen
In message <32ll5mF3mk632U1@individual.net>, Florian Gross <flgr@ccan.de> writes
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk
RSI Information: http://www.objmedia.demon.co.uk/rsi.html
Stephen Kellett wrote:
How does this compare to free solutions like rcov?
I cannot tell you, I have not used rcov. I just tried Google for "rcov Ruby Coverage" and couldn't find anything that I thought was the right target. If you are referring to a tool that produces text output and doesn't give you the output until the tool has ended there is no comparison - both give you coverage results, but one gives you a lot more insight into what is happening and how to represent the results and/or direct your testing, plus providing support for regression testing.
Ah, rcov is a pure Ruby tool that generates a color-highlighted copy of the source code in HTML. I've found it quite useful. It's available via RPA.
Ruby Coverage Validator graphical, stats updated in real time as the app executes - enabling you to direct your testing sequence to ensure maximum coverage if you are running interactively rather than as a regression test. You can run merge results from one run into another, or a central session - ideal for regression testing. You can export the results in HTML or XML. Multiple views onto the same data, etc.
That sounds like it could be useful, though I'm not 100% sure how it would be used in practice.
I especially wonder if you are using trace_funcs (which can be quite slow)
Yes. Although the slowness would be compounded by using a trace func written in Ruby. Our trace func is written in C++. Matz and a few helpful people in this newsgroup provided enough information for us to put things together after examining the source code and writing quite a few test applications.
Interesting -- I've not done this before, but if writing a trace_func in a lower level language speeds it up severely that might be a very important option. Thanks for mentioning this.
Aside: We are not interested in the "Is Python better/worse than Ruby/Java/whatever" debate. It doesn't get anyone anywhere, except distracted from using their own preference in language choice.
That makes a lot of sense, of course. I didn't mean to imply that Ruby or Python were better or that the error was a bad one. I just found it a bit confusing while I was looking for information on how your tool works.
Thank you for the detailed response!
In message <32ll5mF3mk632U1@individual.net>, Florian Gross > <flgr@ccan.de> writes
Ruby Coverage Validator graphical, stats updated in real time as the app executes - enabling you to direct your testing sequence to ensure maximum coverage if you are running interactively rather than as a regression test. You can run merge results from one run into another, or a central session - ideal for regression testing. You can export the results in HTML or XML. Multiple views onto the same data, etc.
That sounds like it could be useful, though I'm not 100% sure how it would be used in practice.
Well, our C++/Java/Python customers seem to find it useful. If you were performing a unit test, you'd probably find it useful to see which lines a particular test executed. If you had all your tests running off a GUI and could run each one on demand, you could view the code coverage results as you executed each command. If the commands were separate Ruby scripts you could view the results in the central session merged into the current session (automatically if you choose) at the end of each script run.
The HTML/XML export allow you to run your regression test suite overnight and come in to work the next day with a session ready to use (in the GUI) and also HTML or XML results for management. We supply XML as well as HTML in case people want to build a custom report for management rather than use the HTML report.
One of our customers (for the C++ product) went from spending 33 hours to perform their regression test with a competing product to 8.5 hours our C++ version of Coverage Validator. Ruby Coverage Validator uses a similar GUI and data collection framework with custom bits for Ruby.
We've added various data export and data visualisation options as people ask for them.
I especially wonder if you are using trace_funcs (which can be quite slow)
Yes. Although the slowness would be compounded by using a trace func written in Ruby. Our trace func is written in C++. Matz and a few helpful people in this newsgroup provided enough information for us to put things together after examining the source code and writing quite a few test applications.
Interesting -- I've not done this before, but if writing a trace_func in a lower level language speeds it up severely that might be a very important option. Thanks for mentioning this.
I thought it would be obvious 
- I can't speak for Ruby users, but the Python people routinely move more expensive operations over to C/C++. I'd expect the same to be true for Ruby.
I haven't done any tests to prove that C/C++ trace function is faster - in order to get the behaviour we wanted we had to use C++ anyway, hence thats the way we did it.
Stephen
In message <32qvlvF3p991aU1@individual.net>, Florian Gross <flgr@ccan.de> writes
--
Stephen Kellett
Object Media Limited http://www.objmedia.demon.co.uk
RSI Information: http://www.objmedia.demon.co.uk/rsi.html
Stephen Kellett wrote:
Ruby Coverage Validator graphical, stats updated in real time as the app executes - enabling you to direct your testing sequence to ensure maximum coverage if you are running interactively rather than as a regression test. You can run merge results from one run into another, or a central session - ideal for regression testing. You can export the results in HTML or XML. Multiple views onto the same data, etc.
That sounds like it could be useful, though I'm not 100% sure how it would be used in practice.
Well, our C++/Java/Python customers seem to find it useful. If you were performing a unit test, you'd probably find it useful to see which lines a particular test executed. If you had all your tests running off a GUI and could run each one on demand, you could view the code coverage results as you executed each command. If the commands were separate Ruby scripts you could view the results in the central session merged into the current session (automatically if you choose) at the end of each script run.
The HTML/XML export allow you to run your regression test suite overnight and come in to work the next day with a session ready to use (in the GUI) and also HTML or XML results for management. We supply XML as well as HTML in case people want to build a custom report for management rather than use the HTML report.
One of our customers (for the C++ product) went from spending 33 hours to perform their regression test with a competing product to 8.5 hours our C++ version of Coverage Validator. Ruby Coverage Validator uses a similar GUI and data collection framework with custom bits for Ruby.
We've added various data export and data visualisation options as people ask for them.
Ah, that's very interesting points. I can see how those features are useful in practice now.
I especially wonder if you are using trace_funcs (which can be quite slow)
Yes. Although the slowness would be compounded by using a trace func written in Ruby. Our trace func is written in C++. Matz and a few helpful people in this newsgroup provided enough information for us to put things together after examining the source code and writing quite a few test applications.
Interesting -- I've not done this before, but if writing a trace_func in a lower level language speeds it up severely that might be a very important option. Thanks for mentioning this.
I thought it would be obvious
Sure, we do that, too. But I always felt that trace_funcs were too slow to have them implemented permanently. That might not be true when writing them in C, however. I'll have a look at that.
In message <32qvlvF3p991aU1@individual.net>, Florian Gross > <flgr@ccan.de> writes