ctools

Buehler Rolf wrote:

Provide uniform high level analysis scripts for common tasks as e.g. spectral points calculation or residual maps. This will start by expanding the obsutils.py script. For later stages the structure of this scripting should be well thought through, it might become the interface to ctools most users use.

to feed the discussion, I think we need a collection of script/classes

• a set of classes/scripts that run the entire analysis (based on configuration files?)
• an ascii reader/writer
• a set of classes that plot the spectrum+point+ul+residual (in plotutils.py or spectrumutils.py)
• a set of classes that plot maps (maybe in the same class than the spectrum one)
• a way to plot LC?
• a way of computing UL (without dependencies)

I am currently working on the plotting tools and I used the tools that Michael Mayer sent us. I thinks the ascii format should be such that the reader knows the units and it convert to the desired units

Sanchez David wrote:

Buehler Rolf wrote:

• an ascii reader/writer

The is the GCsv class for column separated value ASCII files. Can you check whether this satisfies the needs, or is something else needed?

• obsutils.py
  • Observation definition
  • Event simulation
  • Standard ctools analysis workflows

• plotutils.py
  • Stuff needed to display data and results
  • Depends on matplotlib (no further dependency required)

• specutils.py
  • Utilities for spectral analysis, including definition of node functions, splitting runs in energy bins, etc.

• timeutils.py
  • Utilities for temporal analysis, including light curves, GRB analysis

Knödlseder Jürgen wrote:

We have to think how to organize the scripts into modules. So far there is obsutils.py, now we also have plotutils.py. Here some proposal to start the discussion:

• obsutils.py
  • Observation definition
  • Event simulation
  • Standard ctools analysis workflows

• plotutils.py
  • Stuff needed to display data and results
  • Depends on matplotlib (no further dependency required)

• specutils.py
  • Utilities for spectral analysis, including definition of node functions, splitting runs in energy bins, etc.

• timeutils.py
  • Utilities for temporal analysis, including light curves, GRB analysis

I have started working on it and I have a tool that plot SEDs. The rest will follow.

Knödlseder Jürgen wrote:

Sanchez David wrote:

Buehler Rolf wrote:

• an ascii reader/writer

The is the GCsv class for column separated value ASCII files. Can you check whether this satisfies the needs, or is something else needed?

Can this class create a file? I did not see a “write” function

Sanchez David wrote:

Knödlseder Jürgen wrote:

Sanchez David wrote:

Buehler Rolf wrote:

• an ascii reader/writer

The is the GCsv class for column separated value ASCII files. Can you check whether this satisfies the needs, or is something else needed?

Can this class create a file? I did not see a “write” function

I guess not for the moment, but it should of course be able of doing that. For the moment I simply use it for reading Fermi FileFunctions. The class is definitely not very developed. You may create an issue for that.

Knödlseder Jürgen wrote:

Sanchez David wrote:

Knödlseder Jürgen wrote:

Sanchez David wrote:

Buehler Rolf wrote:

• an ascii reader/writer

The is the GCsv class for column separated value ASCII files. Can you check whether this satisfies the needs, or is something else needed?

Can this class create a file? I did not see a “write” function

I guess not for the moment, but it should of course be able of doing that. For the moment I simply use it for reading Fermi FileFunctions. The class is definitely not very developed. You may create an issue for that.

We will have a look so

I forgot to mention, this is also the format in which obsutils.specpoint right now returns spectra. The example file above was then written out with json:

spec = obsutils.specpoints(obssim,“CrabNebula",obsutils.getlogbins())
json.dump(spec,open(“spectrum.json",“w"),indent=4)

I have created a new branch ( if git did what i told me) with a first commit

Sanchez David wrote:

I have created a new branch ( if git did what i told me) with a first commit

Do you have some stuff to be merged in? If yes, please tell me what branch.

I think there is still stuff to check

I also had look at the code and find it very nice to have these functions.
Concerning the light curve, the spectrum and upper limit computation: I would be in favour to have this as individual tools (or cscripts) in ctools. This would be more aligned with the ctools-philosophy. The high-level analysis-tools would then call these classes.

• ctupperlimit (C++)
• cslightcurve (python)
• csspectrum (python, see also #1335)

We can call ctupperlimit within the two csscripts to also get an upper limit in each time - or energy bin.

Note in the meantime, we also added ctbutterfly (see #1051)

Indeed, the overall philosophy is: if you have an atomic task, create a ctool or cscript, if you combine atomic tasks together, or need a nice envelope for a given task, write a Python script.

Mayer Michael wrote:

I would therefore propose to add the following tools:

• ctupperlimit (C++)
• cslightcurve (python)
• csspectrum (python, see also #1335)

• ctlimit or ctulimit
• cslightcrv
• csspec

HI

I’ll take charge of this

thanks

David

thanks David. I created issues #1363, #1364, #1365 for the individual tools. For the spectrum and upper limit, I have some preliminary code available, which I will post in the respective thread.

Mayer Michael wrote:

thanks David. I created issues #1363, #1364, #1365 for the individual tools. For the spectrum and upper limit, I have some preliminary code available, which I will post in the respective thread.

Thanks

I’ll have a look and update your code with mine

David

Well, for LC and bin in energy, I have some concerns. User will want to use CPU farm and make 1 point (of the LC or the spectra) per job no? This is what I do for Fermi analysis.

I am having a look at ctbutterfly and see if I should change my scripts

Well, for LC and bin in energy, I have some concerns. User will want to use CPU farm and make 1 point (of the LC or the spectra) per job no? This is what I do for Fermi analysis.

I agree! The user needs flexibility to launch individual jobs. We could implement a similar logic as in cttsmap, where we added the parameters binmin and binmax. With these parameters the user can specify which range should be computed in the current job.

So I think the new tools should me made in such way that they are calling a tool/script per bin/points (the same for all jobs). In the future, submition to clusters might also be supported (no?)

I had a look a ctbutterfly. It makes the same as my function in plotutils.py
As suggested by Jurgen, I will create a new branch and update the script to use it and also to use the new script (csspec, etc..) when ready.

So I think the new tools should me made in such way that they are calling a tool/script per bin/points (the same for all jobs). In the future, submition to clusters might also be supported (no?)

Since every batch farm works differently, a support for that is (as far as I know) not foreseen in gammalib. However, we could provide easy functionality that the user can specify a part of spectra/maps (individual bins), which should be calculated in the job. Accordingly, the user can submit the same csspec job, just with different binmin, binmax parameters. (Of course you would have to merge afterwards, see, e.g. $CTOOLS/scripts/tsmerge.py)

I had a look a ctbutterfly. It makes the same as my function in plotutils.py

Yes, it is very much inspired by the python script I uploaded a while ago. Of course, there are still some things to improve (#1051).

As suggested by Jurgen, I will create a new branch and update the script to use it and also to use the new script (csspec, etc..) when ready.

Excellent, sounds like a good plan. In my view, we could start with ctulimit, which can then be used in csspec and cslightcrv.

Sanchez David wrote:

Well, for LC and bin in energy, I have some concerns. User will want to use CPU farm and make 1 point (of the LC or the spectra) per job no? This is what I do for Fermi analysis.

A nice alternative to this is the use of OpenMP, which does the parallelization itself and adapts it to the available resources. We had a discussion with Michael on this when he wrote the tsmap tool. Given the fact that processors will get more and more cores in the future, we should really integrate more OpenMP capabilities into the ctools so that the parallelization is transparent to the user. Nevertheless, we still may keep the possibility to perform “external parallelization”.

I was wondering if we could not create a unique tool 'csanalysis’ that run a fit from fits files or GObservation containers. this tools can then be called by the high level analysis tool to create enerby bin or time bin.

what do you think?

I have created a new branch also 1263-high-level-scripts-continued

I don’t quite understand what this tool should do exactly. The name csanalysis seems very generic to me. Do you intend this tool to only run ctlike and csspec afterwards? Or should it give also light curves, maps etc... anything a user can possibly want?

This need to be updated zith the new parameter name and new tools. working on it

Remove analysisutils.py and plotutils.py from the distribution for now. More thinking is needed to define the strategy for common high-level tools.