Bug #1810
csiactcopy breaks when copying a runlist that has more than 68 runs
Status: | Closed | Start date: | 06/28/2016 | |
---|---|---|---|---|
Priority: | Normal | Due date: | ||
Assigned To: | Mayer Michael | % Done: | 100% | |
Category: | - | |||
Target version: | 1.1.0 | |||
Duration: |
Description
I discovered an interesting bug in csiactcopy
:
In case the user provides a runlist that should be copied (instead of the full dataset), the tool issues a command to open the index files only for the given runs. The command looks the following:
selection = "" selection += "["+hduname+"][" for run in self._runs: selection += "OBS_ID=="+str(run) selection += "||" selection = selection[:-2] selection += "]" openstring = filename + selection fits = gammalib.GFits(openstring)
As soon the runlist exceeds 68 entries (both on MacOSX and Scientific Linux), the last command throws an exception with FITS error status 125. Apparently, the
openstring
passed to GFits
becomes simply too long to be processed by cfitsio.
I suppose there are three options to tackle this:
- Throw an error in
csiactcopy
if the runlist exceeds e.g. 60 runs (instruct the user to split list into chunks and runcsiactcopy
several times) - Fix the issue in
csiactcopy
that internally the runlist gets chunked into pieces of maximum 60 runs (might imply some work and testing) - Enhance
GFits
to allow for such long selection strings (e.g. by chunking internally and merging)
I have checked the following page:
https://heasarc.gsfc.nasa.gov/ftools/fv/doc/expressions.html
There it says: “The expression can be an arbitrarily complex series of operations performed on constants, keyword values, and column data taken from the specified FITS TABLE extension. For the case of row filtering, the expression must evaluate to a single boolean value for each row of the table.”
Recurrence
No recurrence.
History
#1 Updated by Mayer Michael over 8 years ago
Update:
I have found this: http://serverfault.com/questions/9546/filename-length-limits-on-linux
It seems the length of a filename is limited to 255 bytes on unix. Could this be the reason for the above error?
#2 Updated by Knödlseder Jürgen over 8 years ago
I found: https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node20.html
Looks like cfitsio limits the maximal length of a filename to 1025 characters. But maybe there are other intrinsic limits.
#3 Updated by Mayer Michael over 8 years ago
Thanks for the link. I guess if it is natively in cfitsio, there is nothing we can do from the gammalib side, right?
This implies we can either throw a meaningful exception (probably more useful for 1.1.0) and add it to a known issues section or try to fix it by internal chunking of the runlist.
#4 Updated by Knödlseder Jürgen over 8 years ago
Looks like it should for the moment be a known issue (maybe you can add this to the doc/source/issues.rst
file?)
Indeed, nothing can be done on the GammaLib side for that problem. In the long run we should probably change csiactcopy
so that it explicitly loops over runs.
#5 Updated by Mayer Michael over 8 years ago
- Status changed from New to Resolved
- Assigned To set to Mayer Michael
- Target version set to 1.1.0
- % Done changed from 0 to 100
Fixed by #1813.
#6 Updated by Knödlseder Jürgen over 8 years ago
- Status changed from Resolved to Closed