Feature #579

Add API Sanity Checker on Jenkins

Added by Deil Christoph over 11 years ago. Updated over 11 years ago.

Status:ClosedStart date:10/30/2012
Priority:NormalDue date:
Assigned To:Knödlseder Jürgen% Done:

100%

Category:-
Target version:-
Duration:

Description

I think adding the API Sanity Checker (or similar tool, I didn’t investigate others) on Jenkins could be useful.
http://ispras.linuxbase.org/index.php/API_Sanity_Checker
It’s a pretty simple tool, basically all it does is run each function / method and check if it crashes or hangs.

I ran it with default options on Gammalib devel 0d0dff7d4c30c1f73e61eadb14a817e463b04e7e and it ran 3964 tests and found 95 SEGV.

This is the first reported error:

    GCTAEventBin* eventbin = new GCTAEventBin();
    eventbin->counts(); //target call

segfaults because
    double*      m_counts;      //!< Pointer to number of counts

is initialized to NULL in the constructor and then dereferenced in the counts method.

Then there are four more error reported for GCTAEventBin which are similar, all due to m_counts == NULL.

Then the next error is this:

    GEnergy* energy = new GEnergy();
    energy->log10keV(); //target call

Here the reason is infinite recursion:
double GEnergy::log10keV(void) const
{
    // Return log10 energy
    return (log10keV()+3.0); 
}

I didn’t further yet, but I think the 95 errors reported are real issues and the 1221 warnings reported are bogus, all the cases I looked at there was an uncaught exception (and the exception was correct).

In the future (when gammalib is widely deployed / used and needs to be stable) I think these related tools by this Russian Linux group could also be useful:
http://upstream-tracker.org/
http://ispras.linuxbase.org/index.php/ABI_compliance_checker
http://pkgdiff.github.com/pkgdiff/

Here are the details how I installed and ran the ABI Sanity Checker (in an Ubuntu VM):

wget https://github.com/downloads/lvc/api-sanity-checker/api-sanity-checker-1.12.10.tar.gz
tar -xzf api-sanity-checker-1.12.10.tar.gz
cd api-sanity-checker-1.12.10
sudo perl Makefile.pl -install --prefix=/usr
cd /tmp
api-sanity-checker -test
firefox test_results/libsample_c/1.0/test_results.html
firefox tests/libsample_cpp/1.0/view_tests.html
mkdir gammalib_sanity
cd gammalib_sanity/
api-sanity-checker -lib libgamma -d VERSION.xml -gen
api-sanity-checker -lib libgamma -d VERSION.xml -build
api-sanity-checker -lib libgamma -d VERSION.xml -run
firefox test_results/libgamma/0.0/test_results.html

I’ll attach the test_results.html report to this issue.

The VERSION.xml file I used is

    <version>0.0</version>
    <headers>/usr/local/gamma/include</headers>
    <libs>/usr/local/gamma/lib</libs>

If it’s not obvious why the SEGV occurred from reading the source it’s easy to debug because each test case is it’s own binary, e.g. the GEnergy::log10keV() infinite recursion:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/gamma/lib
gdb tests/libgamma/0.0/groups/GEnergy/classes/GEnergy/_ZNK7GEnergy8log10keVEv/test
>>> run
>>> tb

Generating and running all tests takes a while (I think about an hour using one core), but doing it nightly or weekly for one build should be easily possible.
I’d be happy to set this up if you want. (I want to learn more how Jenkins works.)

test_results.html Magnifier (4 MB) Deil Christoph, 10/30/2012 12:25 PM


Recurrence

No recurrence.

History

#1 Updated by Knödlseder Jürgen over 11 years ago

Thanks a lot, Christoph. This looks like a pretty powerful tool. I will install it and then work on all the problems it detects.

#2 Updated by Knödlseder Jürgen over 11 years ago

  • Status changed from New to In Progress
  • Assigned To set to Knödlseder Jürgen
  • % Done changed from 0 to 10

Hi Christoph,

I created a new Jenkins job for the API sanity check. Results can be found here: https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/

Ultimately, I may add the relevant steps to the GammaLib code analysis job.

I also started to correct the failures.

#3 Updated by Knödlseder Jürgen over 11 years ago

Investigating the various errors, I found at least one case where the problem is in the test code generated by the sanity checker and not in GammaLib. Here the test code created for the GFits::append(GFitsHDU const& hdu) method:

#include <gammalib/GammaLib.hpp>
    
class GFitsHDU_SubClass: public GFitsHDU
{
public:
    GFitsHDU_SubClass():GFitsHDU(){}
    
    void clear(){}
    
    GFitsHDU* clone() const {
       return (GFitsHDU*) NULL;
    }
    
    void data_close(){}
    
    void data_connect(void* vptr){}
    
    void data_open(void* vptr){}
    
    void data_save(){}
    
    enum GFitsHDU::HDUType exttype() const {
       return GFitsHDU::HT_IMAGE;
    }
    
    std::string print() const {
       return "retval";
    }
};//GFitsHDU_SubClass
    
int main(int argc, char *argv[])
{
    GFits* fits = new GFits();
    GFitsHDU_SubClass hdu;
    fits->append(hdu); //target call
    return 0;
}

This code fails because the clone method implemented in the GFitsHDU_SubClass class returns a NULL pointer while it should return a valid copy of the GFitsHDU_SubClass object.

I guess that the API sanity checker can be tweeked (see http://ispras.linuxbase.org/index.php/Specialized_Type), but this needs a better understanding of how the tool works.

#4 Updated by Knödlseder Jürgen over 11 years ago

  • Status changed from In Progress to Feedback
  • % Done changed from 10 to 100

I went over all errors and corrected those which were weak points of the GammaLib code. The remaining errors are related to bad implementation of test cases. Solving those would require tweeking the setup of the sanity checker.

I change the status of the issue to “Feedback”, as the major issues related to the GammaLib code have been resloved. Improving the sanity checker setup is a “nice to have” feature.

#5 Updated by Deil Christoph over 11 years ago

Is it possible to access the html report for each build?

At https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/ I only see a link to
https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/API_Sanity_Checker_report
which forwards to
https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/API_Sanity_Checker_report/?
and this isn’t the latest report I think.

Here I don’t see a link to access the html report:
https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/6/

#6 Updated by Knödlseder Jürgen over 11 years ago

Hi Christoph,

The links are correct and work on my side. Can’t you see the latest report under https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/API_Sanity_Checker_report? The number of errors is already reduced with respect to your test as I corrected already some of the errors in the devel branch.

#7 Updated by Deil Christoph over 11 years ago

Jürgen Knödlseder wrote:

The links are correct and work on my side. Can’t you see the latest report under https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/API_Sanity_Checker_report? The number of errors is already reduced with respect to your test as I corrected already some of the errors in the devel branch.

I can see https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/API_Sanity_Checker_report with 64 errors.

Is there a way to see the output from previous builds or only the latest?

Why didn’t a build trigger for the latest commit on devel which should have fixed GLATResponseTable::energies ?
Is this a nightly build? Can I see somehow when which builds are supposed to trigger?
https://cta-redmine.irap.omp.eu/projects/gammalib/repository/revisions/a3e90cba1bf837edf3bf4a1eedc22b79b635c6c6

#8 Updated by Knödlseder Jürgen over 11 years ago

Christoph Deil wrote:

Jürgen Knödlseder wrote:

The links are correct and work on my side. Can’t you see the latest report under https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/API_Sanity_Checker_report? The number of errors is already reduced with respect to your test as I corrected already some of the errors in the devel branch.

I can see https://cta-jenkins.irap.omp.eu/job/gammalib-sanity/API_Sanity_Checker_report with 64 errors.

That’s the right one.

Is there a way to see the output from previous builds or only the latest?

I disabled this feature as the test reports are relatively large, so I wanted to prevent filling up the disk space. But I can enable the feature of multiple reports to see how things go.

Why didn’t a build trigger for the latest commit on devel which should have fixed GLATResponseTable::energies ?
Is this a nightly build? Can I see somehow when which builds are supposed to trigger?
https://cta-redmine.irap.omp.eu/projects/gammalib/repository/revisions/a3e90cba1bf837edf3bf4a1eedc22b79b635c6c6

It’s a nightly build. You can only see the trigger conditions in the configuration section of the Job. I may just add a sentence in the description text, so that this information is publicly available.

#9 Updated by Deil Christoph over 11 years ago

Thanks for the infos, now it all makes sense.

To display the reason for each build with a little symbol there is
https://wiki.jenkins-ci.org/display/JENKINS/Build+Trigger+Badge+Plugin
but it doesn’t work at all for me on my localhost Jenkins (didn’t investigate why).

#10 Updated by Deil Christoph over 11 years ago

  • Status changed from Feedback to Closed

#11 Updated by Knödlseder Jürgen over 11 years ago

Christoph Deil wrote:

Thanks for the infos, now it all makes sense.

To display the reason for each build with a little symbol there is
https://wiki.jenkins-ci.org/display/JENKINS/Build+Trigger+Badge+Plugin
but it doesn’t work at all for me on my localhost Jenkins (didn’t investigate why).

Installed and works (maybe it’s a Jenkins version problem on your side?). It only shows the icons for jobs that were started after installation of the plugin.

Also available in: Atom PDF