Updated about 10 years ago by Knödlseder Jürgen

CTA model fitting benchmarks

Model fitting benchmarks are determined using the script benchmark_ml_fitting.py that can be found in the inst/cta/test directory. The script needs to be run in that directory once GammaLib has been installed. It uses test data that are shipped together with the GammaLib package and that serve as a reference. Test data exist for different simulated spatial models, all test data share the following characteristics:
  • observing time of 30 minutes (about 4000 events)
  • power law spectral source model
  • event cube of size 200 x 200 x 20 bins (1.2 arcmin spatial pixel size)
  • energy range 100 GeV-100 TeV
  • cta_dummy_irf response function (performance table)

The following table summarizes the benchmarks results for unbinned, binned and stacked analysis. The response computation differs for these three analysis methods, and consequently, computing times are not identical. Note that the executing time for unbinned analysis scales roughly linearly with observing time (or to be precise with the number of events), while the executing time for stacked analysis is roughly independent of observing time. Execution time for binned analysis scales with the number of individual observations (or runs) that are analysed. Each table cell shows the computing time (i.e. CPU time), the number of fitting iterations needed, the log-likelihood value, and the fitted source parameters given in the order they appear in the model.

The benchmark has been obtained on 30 October 2014 on kepler (CentOS 5, x86_64, AMD Opteron 6164 HE, 1.7 GHz). OpenMP support has been disabled to ensure accurate timing.

Model Unbinned Binned Stacked
Point 0.14 sec (3, 33105.355, 6.037e-16, -2.496) 14.2 sec (3, 17179.086, 5.993e-16, -2.492) 9.1 sec (3, 17179.082, 5.992e-16, -2.492)
Disk 31.4 sec (3, 34176.549, 83.633, 22.014, 0.201, 5.456e-16, -2.459) 252.0 sec (3, 18452.537, 83.632, 22.014, 0.202, 5.394e-16, -2.456) 127.1 sec (3, 18452.503, 83.632, 22.014, 0.202, 5.418e-16, -2.457)
Gauss 36.4 sec (3, 35059.430, 83.626, 22.014, 0.204, 5.374e-16, -2.468) 1539.5 sec (3, 19327.126, 83.626, 22.014, 0.205, 5.346e-16, -2.468) 1237.2 sec (3, 19327.267, 83.626, 22.014, 0.205, 5.368e-16, -2.473)
Shell 70.5 sec (3, 35261.037, 83.633, 22.020, 0.286, 0.115, 5.816e-16, -2.449) 815.6 sec (3, 19301.278, 83.632, 22.019, 0.284, 0.118, 5.771e-16, -2.445) 526.5 sec (4, 19301.380, 83.633, 22.019, 0.286, 0.117, 5.794e-16, -2.447)
Ellipse 156.9 sec (5, 35363.713, 83.569, 21.956, 44.789, 1.998, 0.472, 5.430e-16, -2.482) 5754.1 sec (6, 19943.973, 83.572, 21.956, 44.909, 1.988, 0.474, 5.331e-16, -2.473) 6519.6 sec (6, 19944.471, 83.572, 21.958, 44.927, 2.007, 0.467, 5.314e-16, -2.444)
Diffuse 3.9 sec (18 of which 9 stalled, 32629.409, 5.296e-16, -2.663) 12639.9 sec (18 of which 10 stalled, 18221.681, 5.681e-16, -2.647) 295.8 sec (18 of which 10 stalled, 18221.815, 5.624e-16, -2.665)
The source in the test data is always located at the position of the Crab nebula (RA=83.6331, DEC=22.0145), for the diffuse model, a radio map of the Crab nebula has been used. The nominal parameters for the power law are Prefactor=5.7e-16 and Index=-2.48. The other parameters are (in the order quoted in the table above after RA and DEC):
  • Disk: Radius=0.2
  • Gauss: Sigma=0.2
  • Shell: Radius=0.3, Width=0.1
  • Ellipse: PA=45.0, MajorRadius=2.0, MinorRadius=0.5

Also available in: PDF HTML TXT