Add a science verification script
|Assigned To:||Knödlseder Jürgen||% Done:|
- creates pull distributions for all relevant models
- analyse the pull distribution data and check whether results are compliant with expectation
- prepares a science verification summary report
The number of samples and the observation times used for the pull distributions should be optimized so that the result can be obtained relatively quickly but with enough accuracy to determine whether the result is okay or not.
The results of the summary report should be compiled in a JUnit compatible test report that can be exploited by Jenkins (and possibly Sonar).
The science verification script should be invoked by a simple command, for example
As a requirement, the science verification script should not take longer than a full day of execution time (best would be hours so that it can be run regularly).
The science verification script should be launched at least for every release.
#1 Updated by Knödlseder Jürgen over 6 years ago
- File plaw_prefactor.png added
- File plaw_index.png added
- File disk_ra.png added
- File disk_dec.png added
- File disk_radius.png added
- Status changed from New to In Progress
- Assigned To set to Knödlseder Jürgen
- % Done changed from 0 to 10
Here some plots that show the evolution of the mean and standard deviations in the pull distributions as function of the number of samples. Generally it seems that about 100 samples are needed to reach a mean near 0 and a standard deviation near 1. The plots in the top row are for the spectral parameters of a power law and the plots in the bottom row are for the spatial parameters of a disk model. The analyses were done in unbinned mode and for 1800 secs integration time. A Crab flux has been assumed.
#2 Updated by Knödlseder Jürgen over 6 years ago
- % Done changed from 10 to 50
I added an initial version of the
science_verification.py to the
test directory and also put some test models under the
test/data/sciver directory. The initial script verifies the pull distribution for a power law model and puts the results in a JUnit test report under
The test does 100 trials which needs 4.5 minutes of computing time on Mac OS X 10.6.
#4 Updated by Knödlseder Jürgen over 6 years ago
I now added all spectral and spatial unbinned tests to the script.
I also relaxed a bit the limit conditions on the mean and the standard deviation. The mean now has to be within [-0.4,0.4], the standard deviation has to be within [0.8,1.2]. 100 trials are done for each model.
#6 Updated by Knödlseder Jürgen over 6 years ago
- % Done changed from 80 to 90
Here the test output of the full ctools science verification script (and in parentheses the wall clock time used for the test on a Mac OS X 10.6 2.66 GHz laptop):
******************************* * ctools science verification * ******************************* Test power law model: ......... ok (5 min) Test power law 2 model: ......... ok (5 min) Test exponentially cut off power law model: ........... ok (5 min) Test super exponentially cut off power law model: ............. ok (12 min) Test log parabola model: ........... ok (11 min) Test Gaussian model: ........... ok (6 min) Test file function model: ....... ok (4 min) Test nodes model: ............. ok (5 min) Test point source model: ............. ok (5 min) Test radial disk model: ............... ok (1 hour 6 min) Test radial Gaussian model: ............... ok (1 hour 42 min) Test radial shell model: ................. ok (7 hours 38 min) Test elliptical disk model: ................... ok (3 hours 26 min)