Bug #1810

csiactcopy breaks when copying a runlist that has more than 68 runs

Added by Mayer Michael over 8 years ago. Updated over 8 years ago.

Status:ClosedStart date:06/28/2016
Priority:NormalDue date:
Assigned To:Mayer Michael% Done:

100%

Category:-
Target version:1.1.0
Duration:

Description

I discovered an interesting bug in csiactcopy:
In case the user provides a runlist that should be copied (instead of the full dataset), the tool issues a command to open the index files only for the given runs. The command looks the following:

selection = "" 
selection += "["+hduname+"][" 
for run in self._runs:
    selection += "OBS_ID=="+str(run)
    selection += "||" 
selection = selection[:-2]
selection += "]" 
openstring = filename + selection
fits     = gammalib.GFits(openstring)

As soon the runlist exceeds 68 entries (both on MacOSX and Scientific Linux), the last command throws an exception with FITS error status 125. Apparently, the openstring passed to GFits becomes simply too long to be processed by cfitsio.

I suppose there are three options to tackle this:
  1. Throw an error in csiactcopy if the runlist exceeds e.g. 60 runs (instruct the user to split list into chunks and run csiactcopy several times)
  2. Fix the issue in csiactcopy that internally the runlist gets chunked into pieces of maximum 60 runs (might imply some work and testing)
  3. Enhance GFits to allow for such long selection strings (e.g. by chunking internally and merging)

I have checked the following page:
https://heasarc.gsfc.nasa.gov/ftools/fv/doc/expressions.html
There it says: “The expression can be an arbitrarily complex series of operations performed on constants, keyword values, and column data taken from the specified FITS TABLE extension. For the case of row filtering, the expression must evaluate to a single boolean value for each row of the table.”


Recurrence

No recurrence.

History

#1 Updated by Mayer Michael over 8 years ago

Update:
I have found this: http://serverfault.com/questions/9546/filename-length-limits-on-linux

It seems the length of a filename is limited to 255 bytes on unix. Could this be the reason for the above error?

#2 Updated by Knödlseder Jürgen over 8 years ago

I found: https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node20.html

Looks like cfitsio limits the maximal length of a filename to 1025 characters. But maybe there are other intrinsic limits.

#3 Updated by Mayer Michael over 8 years ago

Thanks for the link. I guess if it is natively in cfitsio, there is nothing we can do from the gammalib side, right?
This implies we can either throw a meaningful exception (probably more useful for 1.1.0) and add it to a known issues section or try to fix it by internal chunking of the runlist.

#4 Updated by Knödlseder Jürgen over 8 years ago

Looks like it should for the moment be a known issue (maybe you can add this to the doc/source/issues.rst file?)

Indeed, nothing can be done on the GammaLib side for that problem. In the long run we should probably change csiactcopy so that it explicitly loops over runs.

#5 Updated by Mayer Michael over 8 years ago

  • Status changed from New to Resolved
  • Assigned To set to Mayer Michael
  • Target version set to 1.1.0
  • % Done changed from 0 to 100

Fixed by #1813.

#6 Updated by Knödlseder Jürgen over 8 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF