Bug #1481

Node function fit results in large difference between observed and modeled Npred value

Added by Knödlseder Jürgen almost 9 years ago. Updated almost 9 years ago.

Status:RejectedStart date:06/20/2015
Priority:NormalDue date:
Assigned To:Knödlseder Jürgen% Done:

100%

Category:-
Target version:1.0.0
Duration:

Description

During a pull distribution test a large offset between observed and modeled Npred value has been observed:

2015-06-19T22:40:19: +======================+
2015-06-19T22:40:19: | Simulate observation |
2015-06-19T22:40:19: +======================+
2015-06-19T22:40:19: === Observation ===
2015-06-19T22:40:19:  Simulation area ...........: 1.9635e+11 cm2
2015-06-19T22:40:19:  Simulation cone ...........: RA=83.63 deg, Dec=22.01 deg, r=5.5 deg
2015-06-19T22:40:19:  Time interval .............: 0 - 1800 s
2015-06-19T22:40:19:  Photon energy range .......: 100 GeV - 100 TeV
2015-06-19T22:40:19:  Event energy range ........: 100 GeV - 100 TeV
2015-06-19T23:04:53:  MC source photons .........: 244136909 [Crab]
2015-06-19T23:04:53:  MC source events ..........: 25435364 [Crab]
2015-06-19T23:04:53:  MC source events ..........: 25435364 (all source models)
2015-06-19T23:05:03:  MC background events ......: 13112
2015-06-19T23:05:03:  MC events .................: 25448476 (all models)
...
2015-06-19T23:08:13: +=================================+
2015-06-19T23:08:13: | Maximum likelihood optimisation |
2015-06-19T23:08:13: +=================================+
2015-06-19T23:14:20:  >Iteration   0: -logL=-84714926.705, Lambda=1.0e-03
2015-06-19T23:20:43:  >Iteration   1: -logL=-84714942.616, Lambda=1.0e-03, delta=15.911, max(|grad|)=20189.418265 [Intensity1:5]
2015-06-19T23:27:19:   Iteration   2: -logL=-84714942.616, Lambda=1.0e-04, delta=-7.027, max(|grad|)=61905.752114 [Intensity1:5] (stalled)
2015-06-19T23:34:02:  >Iteration   3: -logL=-84714942.882, Lambda=1.0e-03, delta=0.266, max(|grad|)=18276.417243 [Intensity1:5]
2015-06-19T23:40:42:   Iteration   4: -logL=-84714942.882, Lambda=1.0e-04, delta=-8.636, max(|grad|)=52339.893580 [Intensity1:5] (stalled)
2015-06-19T23:47:12:   Iteration   5: -logL=-84714942.111, Lambda=1.0e-03, delta=-0.771, max(|grad|)=16566.013235 [Intensity1:5] (stalled)
2015-06-19T23:54:02:   Iteration   6: -logL=-84714942.014, Lambda=1.0e-02, delta=-0.097, max(|grad|)=14950.076140 [Intensity1:5] (stalled)
2015-06-20T00:00:57:   Iteration   7: -logL=-84714942.004, Lambda=1.0e-01, delta=-0.010, max(|grad|)=14906.566158 [Intensity1:5] (stalled)
2015-06-20T00:07:39:   Iteration   8: -logL=-84714942.004, Lambda=1.0e+00, delta=-0.001, max(|grad|)=14903.413753 [Intensity1:5] (stalled)
2015-06-20T00:14:14:   Iteration   9: -logL=-84714942.004, Lambda=1.0e+01, delta=-0.000, max(|grad|)=14905.973680 [Intensity1:5] (stalled)
2015-06-20T00:20:49:   Iteration  10: -logL=-84714942.004, Lambda=1.0e+02, delta=-0.000, max(|grad|)=14906.498404 [Intensity1:5] (stalled)
2015-06-20T00:27:42:   Iteration  11: -logL=-84714942.004, Lambda=1.0e+03, delta=-0.000, max(|grad|)=14906.558966 [Intensity1:5] (stalled)
2015-06-20T00:34:18:   Iteration  12: -logL=-84714942.004, Lambda=1.0e+04, delta=-0.000, max(|grad|)=14906.566163 [Intensity1:5] (stalled)
2015-06-20T00:41:04:  >Iteration  13: -logL=-84714942.004, Lambda=1.0e+05, delta=0.000, max(|grad|)=14906.565672 [Intensity1:5]
2015-06-20T00:47:39: 
2015-06-20T00:47:39: +=========================================+
2015-06-20T00:47:39: | Maximum likelihood optimization results |
2015-06-20T00:47:39: +=========================================+
2015-06-20T00:47:39: === GOptimizerLM ===
2015-06-20T00:47:39:  Optimized function value ..: -84714942.004
2015-06-20T00:47:39:  Absolute precision ........: 0.005
2015-06-20T00:47:39:  Acceptable value decrease .: 2
2015-06-20T00:47:39:  Optimization status .......: converged
2015-06-20T00:47:39:  Number of parameters ......: 11
2015-06-20T00:47:39:  Number of free parameters .: 4
2015-06-20T00:47:39:  Number of iterations ......: 13
2015-06-20T00:47:39:  Lambda ....................: 10000
2015-06-20T00:47:39:  Maximum log likelihood ....: 84714942.004
2015-06-20T00:47:39:  Observed events  (Nobs) ...: 25448476.000
2015-06-20T00:47:39:  Predicted events (Npred) ..: 25447393.506 (Nobs - Npred = 1082.49)

The following XML model had been used;
<?xml version="1.0" standalone="no"?>
<source_library title="source library">
  <source name="Crab" type="PointSource">
    <spectrum type="NodeFunction">
      <node>
        <parameter scale="1.0"   name="Energy"    min="0.1"   max="1.0e20" value="1.0"  free="0"/>
        <parameter scale="1e-07" name="Intensity" min="1e-07" max="1000.0" value="1.0"  free="1"/>
      </node>
      <node>
        <parameter scale="1.0"   name="Energy"    min="0.1"   max="1.0e20" value="10.0" free="0"/>
        <parameter scale="1e-07" name="Intensity" min="1e-07" max="1000.0" value="0.1"  free="1"/>
      </node>
    </spectrum>
    <spatialModel type="SkyDirFunction">
      <parameter free="0" max="360" min="-360" name="RA" scale="1" value="83.6331" />
      <parameter free="0" max="90" min="-90" name="DEC" scale="1" value="22.0145" />
    </spatialModel>
  </source>
  <source name="Background model" type="CTAIrfBackground" instrument="CTA">
    <spectrum type="PowerLaw">
      <parameter name="Prefactor" scale="1.0" value="1.0" min="1e-3" max="1e3"    free="1"/>
      <parameter name="Index"     scale="1.0" value="0.0" min="-5.0" max="+5.0"   free="1"/>
      <parameter name="Scale"     scale="1e6" value="1.0" min="0.01" max="1000.0" free="0"/>
    </spectrum>
  </source>
</source_library>

Recurrence

No recurrence.

History

#1 Updated by Knödlseder Jürgen almost 9 years ago

  • Status changed from New to In Progress
  • Assigned To set to Knödlseder Jürgen
  • Target version set to 1.0.0
  • % Done changed from 0 to 50

It seem that the problem was related to the fact that the XML file defined two nodes at 1 MeV and 10 MeV, hence there was a hugh level arm when extrapolating this to the TeV domain. Also, the intensities chosen led to a large number of events.
Changing the XML file to

<?xml version="1.0" standalone="no"?>
<source_library title="source library">
  <source name="Crab" type="PointSource">
    <spectrum type="NodeFunction">
      <node>
        <parameter scale="1.0e5" name="Energy"    min="0.1"   max="1.0e20" value="1.0"  free="0"/>
        <parameter scale="1e-15" name="Intensity" min="1e-07" max="1000.0" value="8.69" free="1"/>
      </node>
      <node>
        <parameter scale="1.0e8" name="Energy"    min="0.1"   max="1.0e20" value="1.0"  free="0"/>
        <parameter scale="1e-22" name="Intensity" min="1e-07" max="1000.0" value="3.16" free="1"/>
      </node>
    </spectrum>
    <spatialModel type="SkyDirFunction">
      <parameter free="0" max="360" min="-360" name="RA" scale="1" value="83.6331" />
      <parameter free="0" max="90" min="-90" name="DEC" scale="1" value="22.0145" />
    </spatialModel>
  </source>
  <source name="Background model" type="CTAIrfBackground" instrument="CTA">
    <spectrum type="PowerLaw">
      <parameter name="Prefactor" scale="1.0" value="1.0" min="1e-3" max="1e3"    free="1"/>
      <parameter name="Index"     scale="1.0" value="0.0" min="-5.0" max="+5.0"   free="1"/>
      <parameter name="Scale"     scale="1e6" value="1.0" min="0.01" max="1000.0" free="0"/>
    </spectrum>
  </source>
</source_library>

gave a much smaller (though not perfect) Npred difference:
2015-06-20T11:02:41: +======================+
2015-06-20T11:02:41: | Simulate observation |
2015-06-20T11:02:41: +======================+
2015-06-20T11:02:41: === Observation ===
2015-06-20T11:02:41:  Simulation area ...........: 1.9635e+11 cm2
2015-06-20T11:02:41:  Simulation cone ...........: RA=83.63 deg, Dec=22.01 deg, r=5.5 deg
2015-06-20T11:02:41:  Time interval .............: 0 - 1800 s
2015-06-20T11:02:41:  Photon energy range .......: 100 GeV - 100 TeV
2015-06-20T11:02:41:  Event energy range ........: 100 GeV - 100 TeV
2015-06-20T11:02:41:  MC source photons .........: 208050 [Crab]
2015-06-20T11:02:41:  MC source events ..........: 2869 [Crab]
2015-06-20T11:02:41:  MC source events ..........: 2869 (all source models)
2015-06-20T11:02:46:  MC background events ......: 13267
2015-06-20T11:02:46:  MC events .................: 16136 (all models)
...
2015-06-20T11:02:47: +=================================+
2015-06-20T11:02:47: | Maximum likelihood optimisation |
2015-06-20T11:02:47: +=================================+
2015-06-20T11:02:47:  >Iteration   0: -logL=101707.792, Lambda=1.0e-03
2015-06-20T11:02:48:  >Iteration   1: -logL=101707.231, Lambda=1.0e-03, delta=0.562, max(|grad|)=-0.547137 [Prefactor:7]
2015-06-20T11:02:48:  >Iteration   2: -logL=101707.228, Lambda=1.0e-04, delta=0.003, max(|grad|)=0.017482 [Index:8]
2015-06-20T11:02:49: 
2015-06-20T11:02:49: +=========================================+
2015-06-20T11:02:49: | Maximum likelihood optimization results |
2015-06-20T11:02:49: +=========================================+
2015-06-20T11:02:49: === GOptimizerLM ===
2015-06-20T11:02:49:  Optimized function value ..: 101707.228
2015-06-20T11:02:49:  Absolute precision ........: 0.005
2015-06-20T11:02:49:  Acceptable value decrease .: 2
2015-06-20T11:02:49:  Optimization status .......: converged
2015-06-20T11:02:49:  Number of parameters ......: 11
2015-06-20T11:02:49:  Number of free parameters .: 4
2015-06-20T11:02:49:  Number of iterations ......: 2
2015-06-20T11:02:49:  Lambda ....................: 1e-05
2015-06-20T11:02:49:  Maximum log likelihood ....: -101707.228
2015-06-20T11:02:49:  Observed events  (Nobs) ...: 16136.000
2015-06-20T11:02:49:  Predicted events (Npred) ..: 16134.988 (Nobs - Npred = 1.01196)

Note, however, that the relative Npred difference is comparable for both cases (4.2e-5 in the first case compared to 6.3e-5 in the second case).

#2 Updated by Knödlseder Jürgen almost 9 years ago

  • Status changed from In Progress to Rejected
  • % Done changed from 50 to 100

This does not seem to be a bug, just a feature of the XML model used. Issue rejected.

Also available in: Atom PDF