Feature #1555

Add a science verification script

Added by Knödlseder Jürgen about 9 years ago. Updated about 9 years ago.

Status:ClosedStart date:10/15/2015
Priority:NormalDue date:
Assigned To:Knödlseder Jürgen% Done:

100%

Category:-
Target version:-
Duration:

Description

A science verification script should be added that
  • creates pull distributions for all relevant models
  • analyse the pull distribution data and check whether results are compliant with expectation
  • prepares a science verification summary report

The number of samples and the observation times used for the pull distributions should be optimized so that the result can be obtained relatively quickly but with enough accuracy to determine whether the result is okay or not.

The results of the summary report should be compiled in a JUnit compatible test report that can be exploited by Jenkins (and possibly Sonar).

The science verification script should be invoked by a simple command, for example

make science_verification

As a requirement, the science verification script should not take longer than a full day of execution time (best would be hours so that it can be run regularly).

The science verification script should be launched at least for every release.

plaw_prefactor.png (49.4 KB) Knödlseder Jürgen, 10/15/2015 08:54 PM

plaw_index.png (40.5 KB) Knödlseder Jürgen, 10/15/2015 08:54 PM

disk_ra.png (40.3 KB) Knödlseder Jürgen, 10/15/2015 08:54 PM

disk_dec.png (45.8 KB) Knödlseder Jürgen, 10/15/2015 08:54 PM

disk_radius.png (41.6 KB) Knödlseder Jürgen, 10/15/2015 08:54 PM

Plaw_prefactor Plaw_index Disk_ra Disk_dec Disk_radius

Recurrence

No recurrence.

History

#1 Updated by Knödlseder Jürgen about 9 years ago

Here some plots that show the evolution of the mean and standard deviations in the pull distributions as function of the number of samples. Generally it seems that about 100 samples are needed to reach a mean near 0 and a standard deviation near 1. The plots in the top row are for the spectral parameters of a power law and the plots in the bottom row are for the spatial parameters of a disk model. The analyses were done in unbinned mode and for 1800 secs integration time. A Crab flux has been assumed.

#2 Updated by Knödlseder Jürgen about 9 years ago

  • % Done changed from 10 to 50

I added an initial version of the science_verification.py to the test directory and also put some test models under the test/data/sciver directory. The initial script verifies the pull distribution for a power law model and puts the results in a JUnit test report under reports/sciver.xml.

The test does 100 trials which needs 4.5 minutes of computing time on Mac OS X 10.6.

#3 Updated by Knödlseder Jürgen about 9 years ago

  • % Done changed from 50 to 60

I added a target to the Makefile.am that allows to invoke the science verification script by typing:

make science-verification

#4 Updated by Knödlseder Jürgen about 9 years ago

I now added all spectral and spatial unbinned tests to the script.

I also relaxed a bit the limit conditions on the mean and the standard deviation. The mean now has to be within [-0.4,0.4], the standard deviation has to be within [0.8,1.2]. 100 trials are done for each model.

#5 Updated by Knödlseder Jürgen about 9 years ago

  • % Done changed from 60 to 80

#6 Updated by Knödlseder Jürgen about 9 years ago

  • % Done changed from 80 to 90

Here the test output of the full ctools science verification script (and in parentheses the wall clock time used for the test on a Mac OS X 10.6 2.66 GHz laptop):

*******************************
* ctools science verification *
*******************************
Test power law model: ......... ok (5 min)
Test power law 2 model: ......... ok (5 min)
Test exponentially cut off power law model: ........... ok (5 min)
Test super exponentially cut off power law model: ............. ok (12 min)
Test log parabola model: ........... ok (11 min)
Test Gaussian model: ........... ok (6 min)
Test file function model: ....... ok (4 min)
Test nodes model: ............. ok (5 min)
Test point source model: ............. ok (5 min)
Test radial disk model: ............... ok (1 hour 6 min)
Test radial Gaussian model: ............... ok (1 hour 42 min)
Test radial shell model: ................. ok (7 hours 38 min)
Test elliptical disk model: ................... ok (3 hours 26 min)

#7 Updated by Knödlseder Jürgen about 9 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 90 to 100

Merged into devel

Also available in: Atom PDF