Support #1877

Fit results differ from MC input for Prod3 IRFs

Added by Yang Lili over 7 years ago. Updated over 7 years ago.

Status:In ProgressStart date:11/10/2016
Priority:NormalDue date:
Assigned To:Knödlseder Jürgen% Done:

0%

Category:-
Target version:-
Duration:

Description

We are using prod3 baseline and threshold IRFs to run a background simulation.
ctobssim
RA of pointing (degrees) (0-360) [83.63] 266.364
Dec of pointing (degrees) (-90-90) [22.01] -28.9938
Radius of FOV (degrees) (0-180) [5.0]
Start time (MET in s) [0.0]
End time (MET in s) [1800.0] 1800000
Lower energy limit (TeV) [0.1] 0.03
Upper energy limit (TeV) [100.0]
Calibration database [prod2] prod3
Instrument response function [South_0.5h] Baseline_South_50h
Input model XML file [$CTOOLS/share/models/crab.xml] bkg.xml
Output event data file or observation definition XML file [events.fits] bkg_500h.fits

Then we binned the simulated events, with 0.5deg spatial bins and 20 energy bins in the 0.03 -100 TeV energy range.
ctbin
Input event list or observation definition XML file [events.fits] bkg_500h.fits
First coordinate of image center in degrees (RA or galactic l) (0-360) [83.63] 266.364
Second coordinate of image center in degrees (DEC or galactic b) (-90-90) [22.01] -28.9938
Projection method (AIT|AZP|CAR|MER|MOL|STG|TAN) [CAR]
Coordinate system (CEL - celestial, GAL - galactic) (CEL|GAL) [CEL]
Image scale (in degrees/pixel) [0.02] 0.5
Size of the X axis in pixels [200] 10
Size of the Y axis in pixels [200] 10
Algorithm for defining energy bins (FILE|LIN|LOG) [LOG]
Start value for first energy bin in TeV [0.1] 0.03
Stop value for last energy bin in TeV [100.0]
Number of energy bins (1-200) [20]
Output counts cube file [cntcube.fits] bkg_500h_bin.fits

We run a likelihood analysis between a large scale dark matter map (it is not a point source like) and CR simulation maps (CTAIrfBackground) with ctlike.
ctlike
Input event list, counts cube or observation definition XML file [cr_prod3_bl500h.fits] bkg_500h_bin.fits
Calibration database [prod3]
Instrument response function [Baseline_South_50h]
Input model XML file [2TeVdm_cr.xml] 2TeVdm_bkg.xml
Output model XML file [cr_prod3_bl500h_2TeVdm_cr_110.xml] bkg_500h_bin_fit2TeVdm.xml

We get unreasonable results for this step (please see bkg_500h_bin_fit2TeVdm.xml and ctlike.log for our results). But with prod2 we obtained reasonable results with this setup.

We noticed that prod 3 IRFs have less data points (in terms of energy bins etc) — could that be the problem?

Please let us know if you have any advice or have some experience with prod 3 IRFs usage with CTOOLS (other than for standard point source-like analysis).

bkg_500h_bin_fit2TeVdm.xml Magnifier - ctlike results (1 KB) Yang Lili, 11/10/2016 10:42 AM

ctlike.log - log file (7.06 KB) Yang Lili, 11/10/2016 10:42 AM

bkg.xml Magnifier - xml file (526 Bytes) Yang Lili, 11/10/2016 10:43 AM

2TeVdm_bkg.xml Magnifier - dm plus bkg xml file (932 Bytes) Yang Lili, 11/10/2016 10:49 AM

Recurrence

No recurrence.

History

#1 Updated by Yang Lili over 7 years ago

#2 Updated by Knödlseder Jürgen over 7 years ago

I found a strange thing in your XML file:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<source_library title="source library">
  <source name="Darkmatter" type="DiffuseSource">
    <spectrum type="ConstantValue">
      <parameter name="Value" value="0.577642" error="0.0829846" scale="1" min="0" max="1e+08" free="1" />
    </spectrum>
    <spatialModel type="MapCubeFunction" file="DM_mapcube_54_WW_2TeV_csth_Ein_mapcubedldb025deg_dr01_lmax359p75bmax89p75">
      <parameter name="Normalization" value="1" error="0" scale="1" min="0.1" max="10" free="1" />
    </spatialModel>
  </source>
  <source name="CTABackgroundModel" type="CTAIrfBackground" instrument="CTA">
    <spectrum type="PowerLaw">
      <parameter name="Prefactor" value="0.9916" error="0.000370804" scale="1" min="0.001" max="1000" free="1" />
      <parameter name="Index" value="-0.00453959" error="0.000141207" scale="1" min="-5" max="5" free="1" />
      <parameter name="Scale" value="1" scale="1e+06" min="0.01" max="1000" free="0" />
    </spectrum>
  </source>
</source_library>

Both the Value and Normalization are set to free, which leads to degenerate parameters in the model fitting. This explains why ctlike complains as follows:
2016-11-10T09:33:16:    Parameter "Normalization" has zero curvature. Fix parameter.
2016-11-10T09:33:16:  >Iteration   0: -logL=-795146206.405, Lambda=1.0e-03
2016-11-10T09:33:19:  >Iteration   1: -logL=-795146942.609, Lambda=1.0e-03, delta=736.205, max(|grad|)=11919.478645 [Index:4]
2016-11-10T09:33:21:  >Iteration   2: -logL=-795146943.035, Lambda=1.0e-04, delta=0.426, max(|grad|)=92.767487 [Index:4]
2016-11-10T09:33:23:  >Iteration   3: -logL=-795146943.035, Lambda=1.0e-05, delta=0.000, max(|grad|)=1.206720 [Index:4]
2016-11-10T09:33:23:    Free parameter "Normalization" after convergence was reached with frozen parameter.

You may first set the Normalization to free="0".

Then you also have too few energy bins. According to http://cta.irap.omp.eu/ctools/users/issues.html#issue-binned you should at least use 10 bins per spectral decade. You have 3.5 spectral decades from 0.03 - 100 TeV, hence you should use at least 35 bins (and not 20) or even a bit more.

The issue is that the effective area drops quickly when going to lower energies, and you have to make sure that this drop is sufficiently well sampled. So if you want to run an analysis down to 30 GeV you have to increase the number of bins. You can run the analysis for a different number of bins and check the results for each run. With increasing number of bins your result should at some point stabilize (and correspond to the input model within statistics), which is the point where the effective area is sufficiently well sampled.

#3 Updated by Knödlseder Jürgen over 7 years ago

  • Subject changed from prod3 to Fit results differ from MC input for Prod3 IRFs
  • Status changed from New to In Progress
  • Assigned To set to Knödlseder Jürgen

Also available in: Atom PDF