ctools

Hi Iftach,
can you please specify the ctools version and OS you are using?

Why are you attributing the error to ctselect? The message

this = _tools.new_ctlike(*args)
ValueError: *** ERROR in GApplicationPars::operator[](std::string&): Invalid argument. Parameter "chatter" has not been found in parameter file. Please specify a valid parameter name.

Dear Luigi,

Sorry about the confusion. In previous tests (with the same code), I got the error

  File "/afs/ifh.de/group/cta/scratch/sadeh/software/ctools/ctools/lib/python3.6/site-packages/ctools/tools.py", line 1913, in __init__
    this = _tools.new_ctselect(*args)
ValueError: *** ERROR in GApplicationPars::operator[](std::string&): Invalid argument. Parameter "chatter" has not been found in parameter file. Please specify a valid parameter name.

Here is the complete function (It’s part of a larger class, which defines all the input parameters, but you get the deal):

import gammalib
import ctools

def runCtlike(self):
    for n in range(100):
        # get a string, eg 'ENERGY > 0.11'
        cut = foo(n)

        ctselect = ctools.ctselect()
        ctselect['inobs'] = self.conf['evtFileName']
        ctselect["rad"]  = self.conf['rad']
        ctselect["ra"]  = self.conf['ra']
        ctselect["dec"] = self.conf['dec']
        ctselect['tmin'] = self.conf['tmin']
        ctselect['tmax'] = self.conf['tmax']
        ctselect['emin'] = self.conf['e_min']
        ctselect['emax'] = self.conf['e_max']
        ctselect['expr'] = cut

        ctselect.run()
        events = ctselect.obs()

        ctlike = ctools.ctlike(events)
        ctlike['caldb'] = self.conf['caldb']
        ctlike['irf'] = self.conf['irf']
        ctlike['inmodel'] = self.conf['modelFit']
        ctlike['outmodel'] = self.conf['resultFileName']
        ctlike['logfile'] = self.conf['ctlikeLog']
        ctlike['debug'] = self.conf['debug']

        ctlike.execute()

        for model in ctlike.obs().models():
            if model.name() != self.conf['modelFitName']:
                continue

            ts = model.ts()
            # do something with ts ........
    return

The code is running on the batch farm at DESY:

 $ uname -a
Linux wgs18.zeuthen.desy.de 2.6.32-696.20.1.el6.x86_64 #1 SMP Thu Jan 25 08:47:49 CST 2018 x86_64 x86_64 x86_64 GNU/Linux

PS -
I not sure how the par file would come into this.
The loop works file for several iterations (does the fit etc.), and only crashes after a while.
I never directly manipulate the par file, but I don’t know what happens behind the scenes.

Thanks, Luigi.

I have been doing energy/time selection so far, so I don’t think I have a problem with the pointing, but I’ve added you line to the code.

The ticket you referenced confuse me a bit.
I want to run this code on a batch farm, with multiple jobs running in parallel on different output directories. I assume this would imply a possible racing condition with modifying the par file.
How do I define (from the python interface) the directory in which I want the par file for a given job to be stored?

If you do not intend to do any spatial selection your code block should be replaced by

ctselect["rad"] = 'INDEF'

I’m not so familiar with the handling of par files, but I know there is a protection mechanism in place to avoid i/o by multiple processes. A quick and dirty way could be to redefine the environment variable $HOME for each process to point to a different directory, but I’m not sure it would solve your problem and therefore I think it’s best to let Jürgen help you through this.

Great.
Thanks again!

I remember that there was an issue on the DESY batch farm with the file system and par files, Michael Maier run into this problem.

The issue is that there is a file locking mechanism when reading and writing into par files to make sure that no concurrent process is manipulating the par file. Now on the DESY batch farm this locking mechanism does not seem to work.

Nevertheless, in your code I see no parallel processing, so I’m wondering why there should be an issue. My guess is that in your case the ctselect or ctlike par file is empty, or at least the chatter parameter is missing.

Which ctools version are your using? My guess would be that with the development branch your problem will go away, since we changed the way how par files are handled.

Hey Jürgen,

I’m using the dev version (as of July 6th, but I can update again).
I don’t do parallel processing per job, but I am running multiple jobs at once.

Is there a way to set the par file to a local copy/directory for a particular job?
Thanks.

I think I figured it out...
I’ve added the following to the top of my python routine:

new_PFILES_path = 'new_PFILES_path/'
os.makedirs(new_PFILES_path, exist_ok = True)
os.environ['PFILES'] = new_PFILES_path + ';' + os.environ['CTOOLS'] + '/syspfiles'

I’ll run a few tests and close this issue if all is well.
Thank you both for your help.