Feature #1079

Improve GammaLib unit testing experience

Added by Deil Christoph almost 11 years ago. Updated over 10 years ago.

Status:In ProgressStart date:01/13/2014
Priority:NormalDue date:
Assigned To:-% Done:

10%

Category:-
Target version:-
Duration:

Description

(I think we already discussed this a year or two ago, but I couldn’t find an issue for this.)

Currently test-driven development with GammaLib is pretty painful.
(Today Ellis and I tried to make sense of the test failures in #1014 and it was very slow, basically because it was hard to figure out which test_value calls corresponded to which test error)

When a test error occurs the only info printed to the console is in which GammaLib module the error occured:

...
PASS: test_GApplication
FAIL: test_GModel
PASS: test_GSky
PASS: test_GOptimizer
...

The developer than has to find the info at https://cta-redmine.irap.omp.eu/projects/gammalib/wiki/Contributing_to_GammaLib#Running-unit-tests to know to open up the test/reports/GModel.xml file, which looks like this:
https://gist.github.com/cdeil/8406019

Then he has to parse this XML file to find the <failure> element:

    <testcase assertions="" classname="GModel" name="Test GModelPar: Test rescaling: Test if 1 is within 1 +/- 1e-10" status="" time="0" />
    <testcase assertions="" classname="GModel" name="Test GModelPar: Test rescaling: Test if 1 is within 42 +/- 1e-10" status="" time="0">
      <failure message="Value 1 not within 42 +/- 1e-10 (value-expected = -41)." type="" />
    </testcase>
    <testcase assertions="" classname="GModel" name="Test GModelPar: Test rescaling: Test if 1 is within 1 +/- 1e-10" status="" time="0" />

And then he has to search to test/test_GModel.cpp file to find the test case corresponding to Test GModelPar: Test rescaling and then try to figure out which test_value call failed.
(Note: no file name or line number in the XML report file.)

In HESS we are using googletest which provides a much nicer experience.
It tells the developer very clearly on the console where the error occured and what the problem was, see e.g. this introduction:
http://www.ibm.com/developerworks/aix/library/au-googletestingframework.html

Another feature that is super-useful for developers is to be able to select the tests to run:
https://code.google.com/p/googletest/wiki/AdvancedGuide#Selecting_Tests

My suggestion would be to switch to GoogleTest in GammaLib.

It is feature-complete (e.g. it also contains XML test report output as the GammaLib test classes do) and almost completely bug-free (because it’s been tested by millions of users for years) and could e.g. be used for all CTA C++ projects, making test-driven development of CTA software (including GammaLib / ctools) a pleasure.

:-)


Recurrence

No recurrence.

History

#1 Updated by Knödlseder Jürgen almost 11 years ago

The actual test suite was geared towards continuous integration, not the developers :) The actual format is a standard for tools like Jenkins or Sonar, but I agree that it’s not very human readable. We may think about an extension or other option for developers.

#2 Updated by Knödlseder Jürgen almost 11 years ago

  • % Done changed from 0 to 10

I just found out how to run a single unit test. This is pretty easy:

env TESTS="test_python.py" make -e check

runs only the Python test. See https://cta-redmine.irap.omp.eu/projects/gammalib/wiki/Contributing_to_GammaLib#Running-only-a-subset-of-the-unit-tests

#3 Updated by Knödlseder Jürgen almost 11 years ago

  • Status changed from New to In Progress

#4 Updated by Deil Christoph over 10 years ago

By now I have a lot of experience with GoogleTest for C++ (https://code.google.com/p/googletest/) and pytest (http://pytest.org/latest/) for Python and I think implementing this would be the best contribution I can make for Gammalib this summer before the 1.0 release.

I think those are the best packages for C++ / Python and using those would be a huge improvement over the current hand-written Gammalib testing framework:
- Small, can be bundled in the repo (no extra dependency)
- Can be used everywhere in CTA (no extra learning effort for devs)
- Many more features (test writing, running, debugging) and better documentation.

Also available in: Atom PDF