ctools

After running make install I get this:

lfs1> ctbin 
terminate called after throwing an instance of 'GException::par_file_open_error'
  what():  *** ERROR in GPars::read(std::string): Unable to open parameter file '/home/hfm/deil/scratch/software/cta/gcc/install//syspfiles/ctbin.par'. Could not get a lock on the file.
Aborted (core dumped)
lfs1> ls -lh /home/hfm/deil/scratch/software/cta/gcc/install//syspfiles/ctbin.par
-rw-r--r-- 1 deil hfm 2.8K Oct 11 12:30 /home/hfm/deil/scratch/software/cta/gcc/install//syspfiles/ctbin.par
lfs1> which ctbin
~/scratch/software/cta/gcc/install/bin/ctbin
lfs1> gdb ctbin
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying" 
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /lfs/l1/hess/users/deil/scratch/software/cta/gcc/install/bin/ctbin...done.
(gdb) run
Starting program: /lfs/l1/hess/users/deil/scratch/software/cta/gcc/install/bin/ctbin 
[Thread debugging using libthread_db enabled]
terminate called after throwing an instance of 'GException::par_file_open_error'
  what():  *** ERROR in GPars::read(std::string): Unable to open parameter file '/home/hfm/deil/scratch/software/cta/gcc/install//syspfiles/ctbin.par'. Could not get a lock on the file.

Program received signal SIGABRT, Aborted.
0x00007ffff62108a5 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install cfitsio-3.240-3.el6.x86_64 glibc-2.12-1.80.el6_3.5.x86_64 libgcc-4.4.6-4.el6.x86_64 libgomp-4.4.6-4.el6.x86_64 libstdc++-4.4.6-4.el6.x86_64 ncurses-libs-5.7-3.20090208.el6.x86_64 readline-6.0-4.el6.x86_64
(gdb) bt
#0  0x00007ffff62108a5 in raise () from /lib64/libc.so.6
#1  0x00007ffff6212085 in abort () from /lib64/libc.so.6
#2  0x00007ffff6ef3a5d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x00007ffff6ef1be6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00007ffff6ef1c13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x00007ffff6ef1d0e in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00007ffff7ae19ad in GPars::read (this=0x7fffffffdae0, filename="/home/hfm/deil/scratch/software/cta/gcc/install//syspfiles/ctbin.par") at GPars.cpp:899
#7  0x00007ffff7ae2cad in GPars::load (this=0x7fffffffdae0, filename="ctbin.par", args=std::vector of length 1, capacity 1 = {...}) at GPars.cpp:440
#8  0x00007ffff7adafb2 in GApplication::GApplication (this=0x7fffffffda78, name=<value optimized out>, version=<value optimized out>, argc=1, argv=<value optimized out>) at GApplication.cpp:117
#9  0x0000000000404050 in ctbin::ctbin (this=0x7fffffffda70, argc=1, argv=0x7fffffffdd78) at ctbin.cpp:95
#10 0x000000000040392b in main (argc=<value optimized out>, argv=<value optimized out>) at main.cpp:50

I don’t know why ctbin can’t get a lock on the file. The file exists and I can read and write it just fine.
Does the backtrace help figuring out why a core dump occurs?

This is indeed strange. I also have a Scientific Linux 6 installed in our virtual test box, and there it works perfectly. The detailed information of our system is:

SL 6.2
Python     2.6.5
gcc     4.4.4
kernel     2.6.32-71.el6.x86_64
cfitsio     3.240

The locking of the parfiles is a very specific action, that I introduced to allow parallel processing of ctools. Otherwise it could happen that two instances of an executable try to write at the same time to the same file. I had this problem when I was running many jobs in parallel.

I think that the file locking operations are not 100%, so I was indeed looking for cases where they do not work. However, it is weird that this happens on SL which is a quite standard system, and the same type of system that I test here.

I have to think a little more about this. Eventually, I’ll send you a modified GPars.cpp file so that we get some more diagnostics.

I think that the core dump comes from the fact that an exception was thrown, so I’m not really surprised about this.

Jürgen Knödlseder wrote:

I think that the file locking operations are not 100%, so I was indeed looking for cases where they do not work. However, it is weird that this happens on SL which is a quite standard system, and the same type of system that I test here.

We updated to SL 6.3 last week.
But it could also be our file system setup or ...

I think that the core dump comes from the fact that an exception was thrown, so I’m not really surprised about this.

Sorry, I thin we’ve discussed this before, but I don’t remember.
As far as I know every Linux problem should always have a top-level catch and never core dump, except maybe on developer machines?

Jürgen Knödlseder wrote:

The locking of the parfiles is a very specific action, that I introduced to allow parallel processing of ctools. Otherwise it could happen that two instances of an executable try to write at the same time to the same file. I had this problem when I was running many jobs in parallel.

I think that the file locking operations are not 100%, so I was indeed looking for cases where they do not work. However, it is weird that this happens on SL which is a quite standard system, and the same type of system that I test here.

Our sys admin commented that the disk I was using is an old Lustre file system, which does not support file locking.
This is the same system that I plan to use in the future running 100+ parallel ctools jobs.

The ftools / chandra / Fermi software must have a different mechanism for avoiding problems of multiple jobs trying to access the same pfile.
I don’t remember ever seing this locking problem, I did have a problem about /dev/tty when running parallel ftcopy, there the solution was to set this in the submitted scripts:
export HEADASNOQUERY=1