Bug #1521

Shell parameters Width and Radius are biased

Added by Knödlseder Jürgen over 7 years ago. Updated about 7 years ago.

Status:ClosedStart date:08/24/2015
Priority:NormalDue date:
Assigned To:Knödlseder Jürgen% Done:

100%

Category:-
Target version:1.0.0
Duration:

Description

A pull distribution test on the shell model Radius and Width parameters indicates a bias:

This should be fixed before the 1.0.0 release.

shell_radius.png (44.2 KB) Knödlseder Jürgen, 08/24/2015 04:27 PM

shell_width.png (44.4 KB) Knödlseder Jürgen, 08/24/2015 04:27 PM

stacked-shell-DEC.png (37.5 KB) Knödlseder Jürgen, 10/05/2015 11:33 AM

stacked-shell-RA.png (37.3 KB) Knödlseder Jürgen, 10/05/2015 11:33 AM

stacked-shell-radius.png (44.1 KB) Knödlseder Jürgen, 10/05/2015 11:33 AM

stacked-shell-width.png (38.1 KB) Knödlseder Jürgen, 10/05/2015 11:33 AM

Npred-integration.png (6.82 KB) Knödlseder Jürgen, 10/05/2015 12:41 PM

irf6-6_RA.png (36.3 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

irf6-6_DEC.png (36.7 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

irf6-6_Radius.png (37.5 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

irf6-6_Width.png (37.1 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

irf6-6_Prefactor.png (38.4 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

irf6-6_Index.png (37.4 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

w0.05_irf6-6_RA.png (36.3 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

w0.05_irf6-6_DEC.png (36.9 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

w0.05_irf6-6_Radius.png (34.1 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

w0.05_irf6-6_Width.png (33.3 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

w0.05_irf6-6_Prefactor.png (38.2 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

w0.05_irf6-6_Index.png (36.8 KB) Knödlseder Jürgen, 10/09/2015 10:48 AM

disk01.png (33.6 KB) Knödlseder Jürgen, 10/11/2015 09:08 PM

disk02.png (38.2 KB) Knödlseder Jürgen, 10/11/2015 09:08 PM

disk03.png (43.8 KB) Knödlseder Jürgen, 10/11/2015 09:08 PM

disk04.png (38.4 KB) Knödlseder Jürgen, 10/11/2015 09:08 PM

disk05.png (43.5 KB) Knödlseder Jürgen, 10/11/2015 09:08 PM

gauss5_05.png (38.3 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

disk5_01.png (29.6 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

disk5_02.png (38.1 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

disk5_03.png (43.7 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

disk5_04.png (43.8 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

disk5_05.png (38.3 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

gauss5_01.png (38.5 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

gauss5_02.png (38.3 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

gauss5_03.png (38.7 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

gauss5_04.png (43.7 KB) Knödlseder Jürgen, 10/14/2015 12:00 PM

w0.01_irf6-6_Radius.png (37.8 KB) Knödlseder Jürgen, 10/14/2015 12:21 PM

w0.01_irf6-6_Width.png (38.1 KB) Knödlseder Jürgen, 10/14/2015 12:21 PM

w0.02_irf6-6_Radius.png (34.1 KB) Knödlseder Jürgen, 10/14/2015 12:21 PM

w0.02_irf6-6_Width.png (34.1 KB) Knödlseder Jürgen, 10/14/2015 12:21 PM

w0.10_irf6-6_Radius.png (38.4 KB) Knödlseder Jürgen, 10/14/2015 12:33 PM

w0.10_irf6-6_Width.png (37.6 KB) Knödlseder Jürgen, 10/14/2015 12:33 PM

crab_shell_0.3_0.01_1-2TeV.png (42.4 KB) Knödlseder Jürgen, 10/15/2015 12:03 AM

crab_shell_0.28_0.046_1-2TeV.png (42.2 KB) Knödlseder Jürgen, 10/15/2015 12:03 AM

Shell_radius Shell_width Stacked-shell-dec Stacked-shell-ra Stacked-shell-radius Stacked-shell-width Npred-integration Irf6-6_ra Irf6-6_dec Irf6-6_radius Irf6-6_width Irf6-6_prefactor Irf6-6_index W0.05_irf6-6_ra W0.05_irf6-6_dec W0.05_irf6-6_radius W0.05_irf6-6_width W0.05_irf6-6_prefactor W0.05_irf6-6_index Disk01 Disk02 Disk03 Disk04 Disk05 Gauss5_05 Disk5_01 Disk5_02 Disk5_03 Disk5_04 Disk5_05 Gauss5_01 Gauss5_02 Gauss5_03 Gauss5_04 W0.01_irf6-6_radius W0.01_irf6-6_width W0.02_irf6-6_radius W0.02_irf6-6_width W0.10_irf6-6_radius W0.10_irf6-6_width Crab_shell_0.3_0.01_1-2tev Crab_shell_0.28_0.046_1-2tev

Recurrence

No recurrence.

History

#1 Updated by Knödlseder Jürgen over 7 years ago

  • Description updated (diff)

#2 Updated by Knödlseder Jürgen over 7 years ago

Nothing special found so far. I’m wondering whether this is related to an insufficient accuracy in the log-likelihood fit (the absolute precision requirement is set to 0.005).

#3 Updated by Knödlseder Jürgen about 7 years ago

  • Status changed from New to In Progress
  • Assigned To set to Knödlseder Jürgen
  • % Done changed from 0 to 10

#4 Updated by Knödlseder Jürgen about 7 years ago

  • % Done changed from 10 to 20

Knödlseder Jürgen wrote:

Nothing special found so far. I’m wondering whether this is related to an insufficient accuracy in the log-likelihood fit (the absolute precision requirement is set to 0.005).

Setting the precision requirement to 1e-4 did not change the behavior.

So far I tried the following:
  • set iter_phi = 7 (instead of 6) in GCTAResponseIrf::nroi_radial
  • set iter_rho = 7 and 8 (instead of 6) in GCTAResponseIrf::nroi_radial
  • put the shell centre on the ROI centre
  • put the shell 1 degree offset from the ROI centre

None of these modifications remove the bias (i.e. the bias seems to be insensitive to these changes).

Note that the shell radius was 0.3° and the shell width was 0.1° for these examples. Changing the shell radius to 1° and the shell width to 0.5° removed the bias, hence the bias seems to be related to the size of the shell.

#5 Updated by Knödlseder Jürgen about 7 years ago

Another interesting element: there seems to be no bias in the spatial parameters for the stacked analysis (see figures below). This would point towards a problem in the Npred computation.

#6 Updated by Knödlseder Jürgen about 7 years ago

Attached a plot of the Npred integration evaluation that is done for the initial model values. Everything looks reasonable.

#7 Updated by Knödlseder Jürgen about 7 years ago

It seems that the problem is not related to the Npred computation but to the Irf computation. The precision in GCTAResponseIrf::irf_radial is set to 5 iterations in rho and in phi, and increasing the precision to 6 or 7 leads to better results. The ctlike run however has quite a number of stalls, indicating convergence problems.

Probably the reasoning behind is that for small shell widths (e.g. 0.05°) the angular size is comparable (or smaller) to the PSF size, hence the model varies substantially over the integration region (which is set by the PSF size).

#8 Updated by Knödlseder Jürgen about 7 years ago

I now have set the number of iterations in rho and phi to 6. While this leads to convergence problems in ctlike, it gives the correct fitting results.

I did the same for the elliptical and diffuse IRF computations (hope this fixes also #1248).

#9 Updated by Knödlseder Jürgen about 7 years ago

The following histograms show the pull distributions for a shell of radius 0.3° and width of 0.1° (top) and of width 0.05° (bottom). There is still a bias in radius and width for the narrow shell, but note that 0.05° is close to the angular resolution of CTA. There is also a small bias observed in the stacked analysis.

#10 Updated by Knödlseder Jürgen about 7 years ago

#11 Updated by Knödlseder Jürgen about 7 years ago

Knödlseder Jürgen wrote:

I now have set the number of iterations in rho and phi to 6. While this leads to convergence problems in ctlike, it gives the correct fitting results.

I did the same for the elliptical and diffuse IRF computations (hope this fixes also #1248).

I reset the number of iterations to 5 for the diffuse IRF computations as this does not seem to fix the issue #1248. Still have to work on #1248, but this should be done independently.

#12 Updated by Knödlseder Jürgen about 7 years ago

To check the new number of iterations for radial models I generated pull distributions for the disk model with radius 0.01°, 0.02°, 0.03°, 0.04° and 0.05°. This covers the transition between a disk radius that is smaller than the angular resolution to a disk radius that is larger than the angular resolution. Below the respective pull distributions. While the pull distributions for 0.01° and 0.02° are cut at the negative side, the pull distributions from 0.03° on look unbiased.

#13 Updated by Knödlseder Jürgen about 7 years ago

I checked whether the disk and gaussian model also give reasonable results at the limit of the CTA angular resolution with and iteration depth of 5 in rho and phi. The following pull distributions demonstrate that this is the case (top row: disk model, bottom row: gaussian model; sizes are from left to right: 0.01°, 0.02°, 0.03°, 0.04° and 0.05°. Therefore, for non-shell radial models I leave the IRF iteration depth at 5 (leading to faster computation w/r to 6).

#14 Updated by Knödlseder Jürgen about 7 years ago

Below are pull histograms for an IRF iteration depth of 6 and shell widths of 0.01°, 0.02°, 0.03° and 0.05°.

0.01°
0.02°
0.03°
0.05°
0.10°

#17 Updated by Knödlseder Jürgen about 7 years ago

Below are radial profiles of shell models in the 1-2 TeV range. On the left the shell parameters were r=0.3° and w=0.01°, on the right the shell parameters were r=0.28° and w=0.046°. The profiles look very similar. Note that the fit convergence is not very good for shells with narrow width (see excerpt below). Possibly the fit has simply problems finding the minimum.

2015-10-14T12:25:36:  >Iteration   0: -logL=108069.088, Lambda=1.0e-03
2015-10-14T12:25:36:    Parameter "Radius" drives optimization step (step=0.581703)
2015-10-14T12:27:51:   Iteration   1: -logL=108069.088, Lambda=1.0e-03, delta=-1214.245, max(|grad|)=3560.961511 [Radius:2] (stalled)
2015-10-14T12:27:51:    Parameter "Radius" does not drive optimization step anymore.
2015-10-14T12:30:08:   Iteration   2: -logL=108069.088, Lambda=1.0e-02, delta=-1122.984, max(|grad|)=3703.983556 [Radius:2] (stalled)
2015-10-14T12:32:25:   Iteration   3: -logL=108069.088, Lambda=1.0e-01, delta=-39.914, max(|grad|)=3282.352875 [Radius:2] (stalled)
2015-10-14T12:34:38:  >Iteration   4: -logL=108058.279, Lambda=1.0e+00, delta=10.809, max(|grad|)=1385.439966 [Radius:2]
2015-10-14T12:36:51:  >Iteration   5: -logL=108052.492, Lambda=1.0e-01, delta=5.787, max(|grad|)=207.624895 [Radius:2]
2015-10-14T12:39:07:   Iteration   6: -logL=108052.528, Lambda=1.0e-02, delta=-0.036, max(|grad|)=106.863003 [Width:3] (stalled)
2015-10-14T12:41:20:  >Iteration   7: -logL=108052.302, Lambda=1.0e-01, delta=0.226, max(|grad|)=-58.405796 [Radius:2]
2015-10-14T12:43:35:  >Iteration   8: -logL=108052.225, Lambda=1.0e-02, delta=0.077, max(|grad|)=23.955635 [DEC:1]
2015-10-14T12:45:51:   Iteration   9: -logL=108052.375, Lambda=1.0e-03, delta=-0.150, max(|grad|)=64.127946 [Width:3] (stalled)
2015-10-14T12:48:04:  >Iteration  10: -logL=108052.255, Lambda=1.0e-02, delta=0.120, max(|grad|)=37.344665 [DEC:1]
2015-10-14T12:50:18:   Iteration  11: -logL=108052.533, Lambda=1.0e-03, delta=-0.277, max(|grad|)=90.447630 [Width:3] (stalled)
2015-10-14T12:52:35:  >Iteration  12: -logL=108052.284, Lambda=1.0e-02, delta=0.249, max(|grad|)=46.836933 [DEC:1]
2015-10-14T12:54:46:   Iteration  13: -logL=108052.284, Lambda=1.0e-03, delta=-0.396, max(|grad|)=111.588199 [Width:3] (stalled)
2015-10-14T12:57:05:   Iteration  14: -logL=108052.284, Lambda=1.0e-02, delta=-0.011, max(|grad|)=59.407164 [Width:3] (stalled)
2015-10-14T12:59:20:  >Iteration  15: -logL=108052.206, Lambda=1.0e-01, delta=0.078, max(|grad|)=34.066836 [Radius:2]
2015-10-14T13:01:34:  >Iteration  16: -logL=108052.199, Lambda=1.0e-02, delta=0.006, max(|grad|)=23.223715 [Width:3]
2015-10-14T13:03:47:   Iteration  17: -logL=108052.199, Lambda=1.0e-03, delta=-0.042, max(|grad|)=-27.212419 [Width:3] (stalled)
2015-10-14T13:06:00:  >Iteration  18: -logL=108052.190, Lambda=1.0e-02, delta=0.009, max(|grad|)=-15.701380 [Width:3]
2015-10-14T13:08:14:   Iteration  19: -logL=108052.190, Lambda=1.0e-03, delta=-0.045, max(|grad|)=32.002318 [Width:3] (stalled)
2015-10-14T13:10:28:  >Iteration  20: -logL=108052.189, Lambda=1.0e-02, delta=0.001, max(|grad|)=17.863560 [Width:3]

#18 Updated by Knödlseder Jürgen about 7 years ago

Maybe the problem is that for a width that is smaller than the angular resolution the width parameter becomes basically unconstrained and the strong correlation with the radius does not allow to find a good solution.

#19 Updated by Knödlseder Jürgen about 7 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 60 to 100

Below a table of the resulting radius and width parameters as a function of the IRF integration depth. The simulated shell radius was 0.3°, the simulated shell width was 0.01°.

Integration depth Radius Width Iterations -logL
5 0.298800 0.01 9 108008.884
6 0.279862 0.0455535 20 108052.189
7 0.274953 0.053962 18 108054.513
8 0.274230 0.055146 20 108055.228

I thus changed the integration depth in the code to 6. Higher would even be better but the computing time becomes prohibitive for values above 6.

This issue is not fully fixed, but the performance should be acceptable for the first release. I will add a warning in the ctools documentation that the shell width becomes biased (towards larger values) when the width becomes comparable to the angular resolution.

Also available in: Atom PDF