Bug #1370
ctools 00-08-00 'make check' errors and failed test macros
| Status: | In Progress | Start date: | 11/22/2014 | ||
|---|---|---|---|---|---|
| Priority: | High | Due date: | |||
| Assigned To: |  Knödlseder Jürgen | % Done: | 50% | ||
| Category: | - | ||||
| Target version: | - | ||||
| Duration: |
Description
While compiling the latest version of gammalib and ctools (git version from 2014/11/21), everything get installed but ctools does not pass the installation test 'make check’
- ctools unit testing *
***************
Test ctobssim functionality: .......... ok
Test ctobssim on observation container: .... ok
Test ctselect functionality: E.. NOK
Test ctbin functionality: ... ok
Test ctlike functionality: ... ok
Test cttsmap functionality: ... ok
Test ctmodel functionality: ... ok
Test ctskymap functionality: ... ok
Test ctexpcube functionality: ... ok
Test ctpsfcube functionality: ... ok
Test ctbkgcube functionality: ... ok
Test ctcubemask functionality: ... ok
Test ctbutterfly functionality: ... ok
Test unbinned pipeline with FITS file saving: .E.. NOK
Test unbinned in-memory pipeline: .E.. NOK
FAIL: test_python.py ==============================================
1 of 3 tests failed
Please report to jurgen.knodlseder@irap.omp.eu ==============================================
make2: * [check-TESTS] Erreur 1
make2: quittant le répertoire « /raid/cfarn/SOFT/GAMMALIB/20141121/ctools/test »
make1: [check-am] Erreur 2
make1: quittant le répertoire « /raid/cfarn/SOFT/GAMMALIB/20141121/ctools/test »
make: ** [check-recursive] Erreur 1
In addition, some 'examples’ macros fail:
./make_binned_analysis.py- CTA binned analysis scripts *
*****************************
... this script will take 1-2 minutes
Executable analysis pipeline:
=============================
Simulated events (2.96 CPU seconds)
Binned events into counts map (2.56 CPU seconds)
Maximum likelihood fitting (12.52 CPU seconds)
Total wall time elapsed: 7.0 seconds
Total CPU time used ...: 18.04 seconds
In memory analysis pipeline:
============================
Simulated events (2.32 CPU seconds)
Binned events into counts map (2.19 CPU seconds)
Maximum likelihood fitting (18.87 CPU seconds)
Total wall time elapsed: 10.0 seconds
Total CPU time used ...: 23.38 seconds
Make plots (using matplotlib):
==============================
/usr/lib64/python2.6/site-packages/matplotlib/backends/backend_gtk.py:621: DeprecationWarning: Use the new widget gtk.Tooltip
self.tooltips = gtk.Tooltips()
Traceback (most recent call last):
File "./make_binned_analysis.py”, line 422, in <module>
pipeline_v2()
File "./make_binned_analysis.py”, line 232, in pipeline_v2
plot_counts(bin.obs())
File "./make_binned_analysis.py”, line 275, in plot_counts
ebounds = cube.ebounds()
AttributeError: 'SwigPyObject’ object has no attribute 'ebounds’
Snapshot of the faulty section of the code:
# Loop over observations
for obs in observations:
- Get event cube
cube = obs.events()
- Create energy axis
energy = []
ebounds = cube.ebounds()
Now “cube” do not have ebounds attribute anymore and isn’t iterative (next complaint in the script at line 282: for bin in cube:).
ctools.xml 
- make check report file
(9.73 KB)
Recurrence
No recurrence.
History
#1
Updated by Knödlseder Jürgen over 9 years ago
- Status changed from New to In Progress
- Assigned To set to Knödlseder Jürgen
- Target version set to 1.0.0
- % Done changed from 0 to 50
I corrected the make_binned_analysis.py and make_unbinned_analysis.py scripts.
Concerning the test script errors you encountered, could you please post the file test/reports/ctools.xml?
Could you also check if after removing all files in your ~/pfiles folder the make check errors persist?
#2
Updated by Farnier Christian over 9 years ago
- Status changed from In Progress to Closed
- % Done changed from 50 to 100
Thanks for your help. This fully solved the issue.
#3
Updated by Knödlseder Jürgen over 9 years ago
- Status changed from Closed to Feedback
What did you do? Delete the files in the ~/pfiles folder? (we want to keep track of the solutions so that they could go into a “Known Problems” section).
#4
Updated by Farnier Christian over 9 years ago
- File ctools.xml added
Sorry, I was not explicit.
No, I did not deleted the ~/pfiles folder, I wanted to do it step by step. So, I simply updated the ctools directory:
git remote update -p
git pull
and tried again to run the make_binned_analysis.py script. And it works fine now.
As for the report file, I have attached it now. As indeed this is still pending, I shouldn’t have changed the completion level to 100, sorry for the mess.
#5
Updated by Knödlseder Jürgen over 9 years ago
- Status changed from Feedback to Closed
No worries.
Unfortunately the XML file is not very informative, some exceptions occurred. This was maybe indeed due to obsolete pfiles, but they should get updated when we re-run the ctools configuration script (which I guess is done when you connect to a new terminal when you have added the ctools setup script to your .bashrc file).
I think we can close this now.
#6
Updated by Farnier Christian over 9 years ago
Sorry again, I should have said that the fix only applies to the ploting (and therefore access of the energy boundaries), which is the initial problem I reported.
The 'make check’ problem persists, even when removing the pfile/ directory.
For completeness, the source of the problem is the run() function of test_ctselect.py example:
>>> select["infile"].filename("data/crab_events.fits")
>>> select["outfile"].filename("selected_events.fits")
>>> select["ra"].real(83.63)
>>> select["dec"].real(22.01)
>>> select["rad"].real(3.0)
>>> select["tmin"].real(0.0)
>>> select["tmax"].real(1800.0)
>>> select["emin"].real(0.1)
>>> select["emax"].real(100.0)
>>>
>>> select["debug"].boolean(True)
>>> select.run()
2014-11-24T13:56:34: +============+
2014-11-24T13:56:34: | Parameters |
2014-11-24T13:56:34: +============+
2014-11-24T13:56:34: infile ....................: data/crab_events.fits
2014-11-24T13:56:34: outfile ...................: selected_events.fits
2014-11-24T13:56:34: prefix ....................: selected_
2014-11-24T13:56:34: usepnt ....................: no
2014-11-24T13:56:34: ra ........................: 83.629999999999995
2014-11-24T13:56:34: dec .......................: 22.010000000000002
2014-11-24T13:56:34: rad .......................: 3.
2014-11-24T13:56:34: tmin ......................: 0.
2014-11-24T13:56:34: tmax ......................: 1800.
2014-11-24T13:56:34: emin ......................: 0.1
2014-11-24T13:56:34: emax ......................: 100.
2014-11-24T13:56:34: expr ......................:
2014-11-24T13:56:34: usethres ..................: NONE
2014-11-24T13:56:34: chatter ...................: 2
2014-11-24T13:56:34: clobber ...................: yes
2014-11-24T13:56:34: debug .....................: yes
2014-11-24T13:56:34: mode ......................: ql
2014-11-24T13:56:34: logfile ...................: ctselect.log
2014-11-24T13:56:34:
2014-11-24T13:56:34: +===============================+
2014-11-24T13:56:34: | Observations before selection |
2014-11-24T13:56:34: +===============================+
2014-11-24T13:56:34: === GObservations ===
2014-11-24T13:56:34: Number of observations ....: 1
2014-11-24T13:56:34: Number of predicted events : 0
2014-11-24T13:56:34: === GCTAObservation ===
2014-11-24T13:56:34: Name ......................:
2014-11-24T13:56:34: Identifier ................:
2014-11-24T13:56:34: Instrument ................: CTA
2014-11-24T13:56:34: Event file ................: data/crab_events.fits
2014-11-24T13:56:34: Event type ................: EventList
2014-11-24T13:56:34: Statistics ................: Poisson
2014-11-24T13:56:34: Ontime ....................: 1800 s
2014-11-24T13:56:34: Livetime ..................: 1710 s
2014-11-24T13:56:34: Deadtime correction .......: 0.95
2014-11-24T13:56:34: User energy range .........: undefined
2014-11-24T13:56:34: === GCTAPointing ===
2014-11-24T13:56:34: Pointing direction ........: (RA,Dec)=(83.63,22.01)
2014-11-24T13:56:34: Response function .........: undefined
2014-11-24T13:56:34: === GCTAEventList ===
2014-11-24T13:56:34: Number of events ..........: 6141
2014-11-24T13:56:34: Time interval .............: 51544.5 - 51544.5 days
2014-11-24T13:56:34:
2014-11-24T13:56:34: +=================+
2014-11-24T13:56:34: | Event selection |
2014-11-24T13:56:34: +=================+
2014-11-24T13:56:34: === Observation ===
2014-11-24T13:56:35: Time range ................: 0 - 1800 s
2014-11-24T13:56:35: Selected energy range .....: 0.1 - 100 TeV
2014-11-24T13:56:35: Requested ROI .............: Centre(RA,DEC)=(83.63, 22.01) deg, Radius=3 deg
2014-11-24T13:56:35: ROI of data ...............: Centre(RA,DEC)=(83.63, 22.01) deg, Radius=5 deg
2014-11-24T13:56:35: Selected ROI ..............: Centre(RA,DEC)=(83.63, 22.01) deg, Radius=3 deg
2014-11-24T13:56:35: cfitsio selection .........: TIME >= 0,00000000 && TIME <= 1800,00000000 && ENERGY >= 0,10000000 && ENERGY <= 100,00000000 && ANGSEP(83,630000,22,010000,RA,DEC) <= 3,000000
2014-11-24T13:56:35: FITS filename .............: /tmp/filehbtaeQ[EVENTS][TIME >= 0,00000000 && TIME <= 1800,00000000 && ENERGY >= 0,10000000 && ENERGY <= 100,00000000 && ANGSEP(83,630000,22,010000,RA,DEC) <= 3,000000]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/raid/cfarn/SOFT/GAMMALIB/20141121/lib64/python2.6/site-packages/ctools/tools.py", line 408, in run
return _tools.ctselect_run(self)
RuntimeError: *** ERROR in GFits::open(std::string&): Unable to open FITS file "/tmp/filehbtaeQ[EVENTS][TIME >= 0,00000000 && TIME <= 1800,00000000 && ENERGY >= 0,10000000 && ENERGY <= 100,00000000 && ANGSEP(83,630000,22,010000,RA,DEC) <= 3,000000]" (status=431)
I’m not particularly familiar with FITS format, so could it be some misinterpration of dot and comma?
The save() function works just fine.
>>> select.save() ls -lh selected_events.fits -rw-r--r-- 1 cfarn su-users 735K 24 nov. 14:26 selected_events.fits
Opening the file with fv, everything seems perfectly fine:
XTENSION= 'BINTABLE' / binary table extension BITPIX = 8 / 8-bit bytes NAXIS = 2 / 2-dimensional binary table NAXIS1 = 119 / width of table in bytes NAXIS2 = 6141 / number of rows in table PCOUNT = 0 / size of special data area GCOUNT = 1 / one data group (required keyword) TFIELDS = 26 / number of fields in each row TTYPE1 = 'EVENT_ID' / label for field 1 TFORM1 = '1J ' / data format of field: 4-byte INTEGER TZERO1 = 2147483648 / offset for unsigned integers TSCAL1 = 1 / data are not scaled TTYPE2 = 'OBS_ID ' / label for field 2 TFORM2 = '1J ' / data format of field: 4-byte INTEGER TZERO2 = 2147483648 / offset for unsigned integers TSCAL2 = 1 / data are not scaled TTYPE3 = 'TIME ' / label for field 3 TFORM3 = '1D ' / data format of field: 8-byte DOUBLE TTYPE4 = 'TLIVE ' / label for field 4 TFORM4 = '1D ' / data format of field: 8-byte DOUBLE TTYPE5 = 'MULTIP ' / label for field 5 TFORM5 = '1I ' / data format of field: 2-byte INTEGER TTYPE6 = 'TELMASK ' / label for field 6 TFORM6 = '100X ' / data format of field: BIT TTYPE7 = 'RA ' / label for field 7 TFORM7 = '1E ' / data format of field: 4-byte REAL TTYPE8 = 'DEC ' / label for field 8 TFORM8 = '1E ' / data format of field: 4-byte REAL TTYPE9 = 'DIR_ERR ' / label for field 9 TFORM9 = '1E ' / data format of field: 4-byte REAL TTYPE10 = 'DETX ' / label for field 10 TFORM10 = '1E ' / data format of field: 4-byte REAL TTYPE11 = 'DETY ' / label for field 11 TFORM11 = '1E ' / data format of field: 4-byte REAL TTYPE12 = 'ALT ' / label for field 12 TFORM12 = '1E ' / data format of field: 4-byte REAL TTYPE13 = 'AZ ' / label for field 13 TFORM13 = '1E ' / data format of field: 4-byte REAL TTYPE14 = 'COREX ' / label for field 14 TFORM14 = '1E ' / data format of field: 4-byte REAL TTYPE15 = 'COREY ' / label for field 15 TFORM15 = '1E ' / data format of field: 4-byte REAL TTYPE16 = 'CORE_ERR' / label for field 16 TFORM16 = '1E ' / data format of field: 4-byte REAL TTYPE17 = 'XMAX ' / label for field 17 TFORM17 = '1E ' / data format of field: 4-byte REAL TTYPE18 = 'XMAX_ERR' / label for field 18 TFORM18 = '1E ' / data format of field: 4-byte REAL TTYPE19 = 'SHWIDTH ' / label for field 19 TFORM19 = '1E ' / data format of field: 4-byte REAL TTYPE20 = 'SHLENGTH' / label for field 20 TFORM20 = '1E ' / data format of field: 4-byte REAL TTYPE21 = 'ENERGY ' / label for field 21 TFORM21 = '1E ' / data format of field: 4-byte REAL TTYPE22 = 'ENERGY_ERR' / label for field 22 TFORM22 = '1E ' / data format of field: 4-byte REAL TTYPE23 = 'HIL_MSW ' / label for field 23 TFORM23 = '1E ' / data format of field: 4-byte REAL TTYPE24 = 'HIL_MSW_ERR' / label for field 24 TFORM24 = '1E ' / data format of field: 4-byte REAL TTYPE25 = 'HIL_MSL ' / label for field 25 TFORM25 = '1E ' / data format of field: 4-byte REAL TTYPE26 = 'HIL_MSL_ERR' / label for field 26 TFORM26 = '1E ' / data format of field: 4-byte REAL EXTNAME = 'EVENTS ' / name of this extension DSTYP1 = 'TIME ' / Data selection type DSUNI1 = 's ' / Data selection unit DSVAL1 = 'TABLE ' / Data selection value DSREF1 = ':GTI ' / Data selection reference DSTYP2 = 'POS(RA,DEC)' / Data selection type DSUNI2 = 'deg ' / Data selection unit DSVAL2 = 'CIRCLE(83.63,22.01,5)' / Data selection value DSTYP3 = 'ENERGY ' / Data selection type DSUNI3 = 'TeV ' / Data selection unit DSVAL3 = '0.1:100 ' / Data selection value NDSKEYS = 3 / Number of data selections CREATOR = 'GammaLib' / Program which created the file TELESCOP= 'CTA ' / Telescope OBS_ID = 0 / Observation identifier DATE_OBS= '2000-01-01' / Observation start date TIME_OBS= '11:58:56' / Observation start time DATE_END= '2000-01-01' / Observation end date TIME_END= '12:28:56' / Observation end time TSTART = 0 / [s] Mission time of start of observation TSTOP = 1 / [s] Mission time of end of observation MJDREFI = 51544 / [days] Integer part of time reference MJD MJDREFF = 5 / [days] Fractional part of time reference MJD TIMEUNIT= 's ' / Time unit TIMESYS = 'TT ' / Time system TIMEREF = 'LOCAL ' / Time reference TELAPSE = 1 / [s] Mission elapsed time ONTIME = 1 / [s] Total good time including deadtime LIVETIME= 1 / [s] Total livetime DEADC = 9 / Deadtime correction factor TIMEDEL = 1 / Time resolution OBJECT = ' ' / Observed object RA_OBJ = 0 / [deg] Target Right Ascension DEC_OBJ = 0 / [deg] Target Declination RA_PNT = 8 / [deg] Pointing Right Ascension DEC_PNT = 2 / [deg] Pointing Declination ALT_PNT = 0 / [deg] Average altitude of pointing AZ_PNT = 0 / [deg] Average azimuth of pointing RADECSYS= 'FK5 ' / Coordinate system EQUINOX = 2 / Epoch CONV_DEP= 0 / Convergence depth of telescopes CONV_RA = 0 / [deg] Convergence Right Ascension CONV_DEC= 0 / [deg] Convergence Declination OBSERVER= 'string ' / Observer N_TELS = 100 / Number of telescopes in event list TELLIST = 'string ' / Telescope IDs GEOLAT = 0 / [deg] Geographic latitude of array centre GEOLON = 0 / [deg] Geographic longitude of array centre ALTITUDE= 0 / [km] Altitude of array centre EUNIT = 'TeV ' / Energy unit EVTVER = 'draft1 ' / Event list version number END
#7
Updated by Knödlseder Jürgen over 9 years ago
- Status changed from Closed to In Progress
- % Done changed from 100 to 50
This is VERY strange. Here the code that writes the time selection string:
// Format time with sufficient accuracy and add to selection string
char cmin[80];
char cmax[80];
sprintf(cmin, "%.8f", tmin);
sprintf(cmax, "%.8f", tmax);
selection = "TIME >= "+std::string(cmin)+" && TIME <= "+std::string(cmax);
if (logTerse()) {
log << gammalib::parformat("Time range");
log << tmin << " - " << tmax << " s" << std::endl;
}
It looks like if the
sprintf(cmin, "%.8f", tmin) writes a comma instead of a dot.
Do you use a custom compiled version of ctools? I guess you have selected French in your OS?
This apparently has happened to others already:- http://stackoverflow.com/questions/3709886/sprintf-commas-and-dots-in-c-and-localization
- http://stackoverflow.com/questions/14753505/c-c-printf-use-commas-instead-of-dots-as-decimal-separator
- http://en.cppreference.com/w/cpp/locale/setlocale
We may need to add some code to cools to make this country safe!!!
#8
Updated by Farnier Christian over 9 years ago
I am actually on a swedish system...
Also, I haven’t changed my setup and everything was ok for the previous version of gammalib/ctools.
#9
Updated by Knödlseder Jürgen over 9 years ago
This part of the code has probably been changed. Any clues if a Swedish system uses comma or dot as separator?
#10
Updated by Knödlseder Jürgen over 9 years ago
Well, looks like Sweden also uses a comma: http://en.wikipedia.org/wiki/Decimal_mark
#11
Updated by Knödlseder Jürgen over 8 years ago
Christian, is this still a problem for you or have you fixed the issue?
#12
Updated by Knödlseder Jürgen over 8 years ago
- Priority changed from Normal to Low
- Target version deleted (
1.0.0)
Nothing new on that, and since about almost one year nobody has bothered about that, so let’s keep it as an issue but remove it from the release 1.0.
#13
Updated by Knödlseder Jürgen about 7 years ago
- Priority changed from Low to High
Eirik Saether Hatlen stumbled possibly over the same problem. He wrote:
I have started using CTOOLS and was making reasonable unbined .fits file using ctselect for my studies.
All of a sudden I get the following error when I enter ctselect in the terminal:
libc++abi.dylib: terminating with uncaught exception of type GException::par_file_syntax_error: *** ERROR in GApplicationPars::parse(): Syntax error occured in the following line of the parameter file (found 8 fields, require 7): rad, r, a,0.5,1,0,180, "Radius of ROI (degrees)" Abort trap: 6
I dont recall doing anything other than calling ctselect. All the other functions are working (i.e. ctobssim, ctbin etc).
Any suggestions? CTOOLS was installed completely standard by the instructions on MAC OS X El Capitan 10.11.6
Best
Eirik