Feature #1938

pickle gammalib/ctools objects

Added by Kelley-Hoskins Nathan over 7 years ago. Updated almost 6 years ago.

Status:ClosedStart date:03/01/2017
Priority:NormalDue date:
Assigned To:Knödlseder Jürgen% Done:

100%

Category:-
Target version:1.6.0
Duration:

Description

I’m in a situation where pickling an object to a file would be greatly helpful, but it seems swig objects don’t want to be pickled. The test script below illustrates this.

import gammalib, pickle
a = gammalib.GEnergy(1,'TeV')
with open('test.pickle','wb') as f :
  pickle.dump( a, f, pickle.HIGHEST_PROTOCOL )

results in

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
TypeError: can't pickle SwigPyObject objects

From this link:
http://stackoverflow.com/questions/9310053/how-to-make-my-swig-extension-module-work-with-pickle
it looks like the only thing missing is __setstate__ and __getstate__ in the swig interface. However, I think each class would require a unique pair of functions, so its not a small task.

Recurrence

No recurrence.

History

#1 Updated by Knödlseder Jürgen over 7 years ago

Funny, I stumbled over exactly the same problem today ;)

I agree that pickling should be made possible.

#2 Updated by Kelley-Hoskins Nathan over 7 years ago

I guess also working with json would be nice as well. I find that I often need to read/write GEnergy and GSkyDir to a text file with json, so I have to resort to simpler conversions. I’d be interested in working on this feature once I’ve submitted my thesis in a few weeks.

#3 Updated by Knödlseder Jürgen over 6 years ago

  • Status changed from New to In Progress
  • Assigned To set to Knödlseder Jürgen
  • Target version set to 1.6.0
  • % Done changed from 0 to 10

I started to implement the __getstate__ and __setstate__ methods for all Python classes. To make sure that the methods work as expected, pickeling tests were added to the unit tests.

So far the xml module is complete, and the obs module is complete with the exception of the GCaldb class

I also started to work on the cta module, the support module, the sky module, the models module and the opt module.

#4 Updated by Knödlseder Jürgen over 6 years ago

  • % Done changed from 10 to 20
The following modules are now fully pickable:
  • fits
  • obs
  • xml
Some classes of the following modules are already pickable:
  • model
  • opt
  • sky
  • inst/cta
The following modules were not yet treated:
  • app
  • base (not sure that we need to treat this module)
  • linalg
  • numerics
  • support
  • template (should add pickability to templates for the future)
  • test (not sure that we need to treat this module)
  • vo
  • xspec

#5 Updated by Knödlseder Jürgen over 6 years ago

  • % Done changed from 20 to 30
The following modules are now fully pickable:
  • fits
  • obs
  • model
  • xml
Some classes of the following modules are already pickable:
  • opt
  • sky
  • inst/cta
The following modules were not yet treated:
  • app
  • base (not sure that we need to treat this module)
  • linalg
  • numerics
  • support
  • template (should add pickability to templates for the future)
  • test (not sure that we need to treat this module)
  • vo
  • xspec

#6 Updated by Knödlseder Jürgen over 6 years ago

As a hint, the following code may be used to write successive FITS files for bit-to-bit comparison:

i = 0
while gammalib.GFilename('test_pha_pickeling%d.fits' % i).exists():
    i += 1
fits.saveto('test_pha_pickeling%d.fits' % i)

and the following command provides a bit-to-bit comparison in the terminal
$ diff -y <(xxd test/test_pha_pickeling2.fits) <(xxd test/test_pha_pickeling3.fits)

#7 Updated by Knödlseder Jürgen over 6 years ago

  • % Done changed from 30 to 40
The following modules are now fully pickable:
  • fits
  • obs
  • model
  • sky
  • xml
  • xspec
Some classes of the following modules are already pickable:
  • opt
  • inst/cta (almost finished)
The following modules were not yet treated:
  • app
  • base (not sure that we need to treat this module)
  • linalg
  • numerics
  • support
  • template (should add pickability to templates for the future)
  • test (not sure that we need to treat this module)
  • vo
  • inst/lat
  • inst/com
  • inst/mwl

#8 Updated by Knödlseder Jürgen over 6 years ago

  • % Done changed from 40 to 50

The CTA module is now completed, except of performance table and vector IRF formats that are not used anymore, and the cube internal classes that are unlikely pickled. So for most use cases, pickeling is now fully supported.

I now continue with the implementation of pickeling in the app module so that pickeling of cscripts will become possible.

#9 Updated by Knödlseder Jürgen over 6 years ago

  • % Done changed from 50 to 60
The following modules are now fully pickable:
  • fits
  • obs
  • model
  • sky
  • xml
  • xspec
Some classes of the following modules are already pickable:
  • app (only GLog class missing)
  • opt
  • inst/cta (almost finished)
The following modules were not yet treated:
  • base (not sure that we need to treat this module)
  • linalg
  • numerics
  • support
  • template (should add pickability to templates for the future)
  • test (not sure that we need to treat this module)
  • vo
  • inst/lat
  • inst/com
  • inst/mwl

#10 Updated by Knödlseder Jürgen over 6 years ago

  • % Done changed from 60 to 70

I implemented pickling on the ctools side. I added pickling unit tests for cscaldb, cslightcrv and cstsmapsplit to make sure that pickling works for the three base classes cscript, csobservation and cslikelihood.

This means that multiprocessing should now be possible.

The code is in the integration branch since I’m still doing the integration tests.

#11 Updated by Knödlseder Jürgen over 6 years ago

Code is now in devel. So far there are no __getstate__ and __setstate__ methods in the cscripts, but they are eventually needed to make sure that all class members are properly handled (not sure about this).

#12 Updated by Knödlseder Jürgen over 6 years ago

The following modules are now fully pickable:
  • fits
  • obs
  • opt
  • model
  • sky
  • xml
  • xspec
  • inst/mwl
  • template
The following modules are considered complete, although not fully pickable:
  • inst/com (not pickable: GCOMIaq, GCOMInstChars, GCOMSelection, GCOMStatus)
  • inst/cta (not pickable: GCTAAeffArf, GCTAAeffPerfTable, GCTABackgroundPerfTable, GCTACubeSourceDiffuse, GCTACubeSourcePoint, GCTAEdispPerfTable, GCTAEdispRmf, GCTAPsfPerfTable, GCTAPsfVector)
  • inst/lat (not pickable: GLATEventAtom, GLATEventBin, GLATResponse)
  • app (not pickable: GLog)
Some classes of the following modules are already pickable:
  • ...
The following modules were not yet treated:
  • base (not sure that we need to treat this module)
  • linalg
  • numerics
  • support
  • test (not sure that we need to treat this module)
  • vo

#13 Updated by Knödlseder Jürgen almost 6 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 70 to 100

For the time being, the pickling support is sufficient.

Also available in: Atom PDF