Support #2784
compile gammalib/ctools against anaconda python
| Status: | Closed | Start date: | 01/12/2019 | |
|---|---|---|---|---|
| Priority: | Normal | Due date: | ||
| Assigned To: |  Knödlseder Jürgen | % Done: | 100% | |
| Category: | - | |||
| Target version: | 1.6.0 | |||
| Duration: |
Description
I’m in a situation where I can get access to ~1000 cpu cores, but to do so I need to compile gammalib and ctools against anaconda python. The situation is:
- I have my own veritas python package that depends on 10ish other anaconda python packages, so I can’t go without anaconda
- I need to manually compile gammalib and ctools devel, which have a bugfix that the anaconda gammalib releases dont have yet, so I can’t use anaconda gammalib/ctools
When I configure+make gammalib against the system python with the following command, leaving out all anaconda stuff, gammalib+ctools compile fine.
./configure --prefix=$PWD LDFLAGS=-L$SOFTWARE/cfitsio/lib CPPFLAGS=-I$SOFTWARE/cfitsio/include/
When I try to make this with anaconda python, I get many errors like:
...
creating build/temp.linux-x86_64-3.6/gammalib
gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I../include -I/nv/hp11/nkelleyh3/data/software/cfitsio/include/ -I/gpfs/pace2/project/pc1/nkelleyh3/software/miniconda3/include/python3.6m -I/gpfs/pace2/project/pc1/nkelleyh3/software/miniconda3/include/python3.6m -I../inst/mwl/include -I../inst/cta/include -I../inst/lat/include -I../inst/com/include -I/nv/hp11/nkelleyh3/data/software/miniconda3/include/python3.6m -c gammalib/app_wrap.cpp -o build/temp.linux-x86_64-3.6/gammalib/app_wrap.o -fopenmp
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
gammalib/app_wrap.cpp: In function ‘swig_module_info* SWIG_Python_GetModule()’:
gammalib/app_wrap.cpp:2452:51: error: ‘PyCObject_Import’ was not declared in this scope
(char*)"type_pointer" SWIG_TYPE_TABLE_NAME);
^
gammalib/app_wrap.cpp: In function ‘void SWIG_Python_SetModule(swig_module_info*)’:
gammalib/app_wrap.cpp:2521:92: error: ‘PyCObject_FromVoidPtr’ was not declared in this scope
PyObject *pointer = PyCObject_FromVoidPtr((void *) swig_module, SWIG_Python_DestroyModule);
^
gammalib/app_wrap.cpp:2512:22: warning: unused variable ‘swig_empty_runtime_method_table’ [-Wunused-variable]
static PyMethodDef swig_empty_runtime_method_table[] = { {NULL, NULL, 0, NULL} };/* Sentinel */
^
gammalib/app_wrap.cpp: In function ‘swig_type_info* SWIG_Python_TypeQuery(const char*)’:
gammalib/app_wrap.cpp:2544:60: error: ‘PyCObject_AsVoidPtr’ was not declared in this scope
descriptor = (swig_type_info *) PyCObject_AsVoidPtr(obj);
^
gammalib/app_wrap.cpp:2549:51: error: ‘PyCObject_FromVoidPtr’ was not declared in this scope
obj = PyCObject_FromVoidPtr(descriptor, NULL);
^
...
gammalib/app_wrap.cpp:17046:59: error: ‘PyString_AsString’ was not declared in this scope
temp2 = GFilename(std::string(PyString_AsString(obj1)));
^
gammalib/app_wrap.cpp:17050:106: error: ‘PyString_AsString’ was not declared in this scope
temp2 = GFilename(std::string(PyString_AsString(PyUnicode_AsEncodedString(obj1, "utf-8", "Error ~"))));
^
gammalib/app_wrap.cpp: In function ‘PyObject* _wrap_GApplicationPar_filename(PyObject*, PyObject*)’:
gammalib/app_wrap.cpp:17847:35: error: ‘PyString_Check’ was not declared in this scope
if (PyString_Check(argv[1]) ||
^
error: command 'gcc' failed with exit status 1
make[3]: *** [build] Error 1
make[3]: Leaving directory `/gpfs/pace2/project/pc1/nkelleyh3/software/gammalib-devel/pyext'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/gpfs/pace2/project/pc1/nkelleyh3/software/gammalib-devel/pyext'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/gpfs/pace2/project/pc1/nkelleyh3/software/gammalib-devel'
make: *** [all] Error 2
I’ve found python has a python-config, which spits out various flags, but anaconda version doesn’t symlink it very well:
$ cd /nv/hp11/nkelleyh3/data/software/miniconda3/bin $ ls -l py*config lrwxrwxrwx 1 nkelleyh3 phy-otte 17 Jan 6 08:19 python3.6-config -> python3.6m-config -rwxrwxr-x 1 nkelleyh3 phy-otte 3428 Jan 6 08:18 python3.6m-config lrwxrwxrwx 1 nkelleyh3 phy-otte 17 Jan 6 08:19 python3-config -> python3.6m-config
I tried adding the include flags from this python-config into the gammalib configure command, like:
./configure --prefix=$PWD LDFLAGS=-L$SOFTWARE/cfitsio/lib CPPFLAGS="-I$SOFTWARE/cfitsio/include/ $(python3.6m-config --includes)"
But I get the same above errors. I’ve also added miniconda/lib to LD_LIBRARY_PATH, but no luck that way either.
I realize that if this build is successful, and anaconda updates any part of itself, this gammalib/ctools will probably break and need to be recompiled, but thats acceptable price for 1000 cpu cores.
Does anyone know the right configure commands to compile gammalib/ctools against anaconda python?
Recurrence
No recurrence.
History
#1
Updated by Knödlseder Jürgen over 5 years ago
Here is how GammaLib and ctools are built for anaconda. You may check if this also works for you:
# Setup environment export PATH="$HOME/miniconda3/bin:$PATH" source activate build # Clean conda source and build cache conda clean -s rm -rf $HOME/miniconda3/conda-bld/*-64/* rm -rf $HOME/miniconda3/conda-bld/src_cache/* # Create conda recipe ./autogen.sh ./configure # Build conda packages conda-build dev/conda.recipe
# Setup environment export PATH="$HOME/miniconda3/bin:$PATH" source activate build # Clean conda source and build cache conda clean -s rm -rf $HOME/miniconda3/conda-bld/*-64/* rm -rf $HOME/miniconda3/conda-bld/src_cache/* # Create conda recipe export GAMMALIB=$HOME/jenkins/install/release/gammalib ./autogen.sh ./configure # Build conda packages conda-build dev/conda.recipe
#2
Updated by Kelley-Hoskins Nathan over 5 years ago
- Status changed from New to Resolved
I’ve got it working. I think the trick is that gcc, swig, and cfitsio must all be installed via anaconda, rather than using other (system or manual) installations.
The commands I used to install them are:
$ conda install gcc swig $ conda install -c conda-forge cfitsio
Then configuring gammalib and ctools was just like normal (no extra lib or include flags):
$ cd gammalib $ ./autogen.sh $ ./configure --prefix=$PWD $ make $ make install
My current package list is (most packages came by default with anaconda):
# packages in environment at /nv/hp11/nkelleyh3/data/software/anaconda: # # Name Version Build Channel _ipyw_jlab_nb_ext_conf 0.1.0 py36_0 alabaster 0.7.12 py36_0 anaconda custom py36hbbc8b67_0 anaconda-client 1.7.2 py36_0 anaconda-navigator 1.9.6 py36_0 anaconda-project 0.8.2 py36_0 asn1crypto 0.24.0 py36_0 astroid 2.1.0 py36_0 astropy 3.1.1 py36h7b6447c_0 atomicwrites 1.2.1 py36_0 attrs 18.2.0 py36h28b3542_0 babel 2.6.0 py36_0 backcall 0.1.0 py36_0 backports 1.0 py36_1 backports.os 0.1.1 py36_0 backports.shutil_get_terminal_size 1.0.0 py36_2 beautifulsoup4 4.6.3 py36_0 bitarray 0.8.3 py36h14c3975_0 bkcharts 0.2 py36_0 blas 1.0 mkl blaze 0.11.3 py36_0 bleach 3.0.2 py36_0 blosc 1.14.4 hdbcaa40_0 bokeh 1.0.3 py36_0 boto 2.49.0 py36_0 bottleneck 1.2.1 py36h035aef0_1 bzip2 1.0.6 h14c3975_5 ca-certificates 2018.11.29 ha4d7672_0 conda-forge cairo 1.14.12 h8948797_3 certifi 2018.11.29 py36_1000 conda-forge cffi 1.11.5 py36he75722e_1 cfitsio 3.430 hc04bd9f_1001 conda-forge chardet 3.0.4 py36_1 click 7.0 py36_0 cloog 0.18.0 0 cloudpickle 0.6.1 py36_0 clyent 1.2.2 py36_1 colorama 0.4.1 py36_0 conda 4.5.12 py36_1000 conda-forge conda-build 3.17.6 py36_0 conda-env 2.6.0 1 conda-verify 3.1.1 py36_0 contextlib2 0.5.5 py36_0 cryptography 2.3.1 py36hdffb7b8_0 conda-forge cryptography-vectors 2.3.1 py36_1000 conda-forge curl 7.63.0 h74213dd_0 conda-forge cycler 0.10.0 py36_0 cython 0.29.2 py36he6710b0_0 cytoolz 0.9.0.1 py36h14c3975_1 dask 1.0.0 py36_0 dask-core 1.0.0 py36_0 datashape 0.5.4 py36_1 dbus 1.13.2 h714fa37_1 decorator 4.3.0 py36_0 defusedxml 0.5.0 py36_1 distributed 1.25.2 py36_0 docutils 0.14 py36_0 entrypoints 0.2.3 py36_2 et_xmlfile 1.0.1 py36_0 expat 2.2.6 he6710b0_0 fastcache 1.0.2 py36h14c3975_2 filelock 3.0.10 py36_0 flask 1.0.2 py36_1 flask-cors 3.0.7 py36_0 fontconfig 2.13.0 h9420a91_0 freetype 2.9.1 h8a8886c_1 fribidi 1.0.5 h7b6447c_0 future 0.17.1 py36_0 gcc 4.8.5 7 get_terminal_size 1.0.0 haa9412d_0 gevent 1.4.0 py36h7b6447c_0 glib 2.56.2 hd408876_0 glob2 0.6 py36_1 gmp 6.1.2 h6c8ec71_1 graphite2 1.3.13 h23475e2_0 greenlet 0.4.15 py36h7b6447c_0 gst-plugins-base 1.14.0 hbbd80ab_1 gstreamer 1.14.0 hb453b48_1 h5py 2.9.0 py36h7918eee_0 harfbuzz 1.8.8 hffaf4a1_0 hdf5 1.10.4 hb1b8bf9_0 heapdict 1.0.0 py36_2 html5lib 1.0.1 py36_0 icu 58.2 h9c2bf20_1 idna 2.8 py36_0 imageio 2.4.1 py36_0 imagesize 1.1.0 py36_0 importlib_metadata 0.6 py36_0 intel-openmp 2019.1 144 ipykernel 5.1.0 py36h39e3cac_0 ipython 7.2.0 py36h39e3cac_0 ipython_genutils 0.2.0 py36_0 ipywidgets 7.4.2 py36_0 isl 0.12.2 0 isort 4.3.4 py36_0 itsdangerous 1.1.0 py36_0 jbig 2.1 hdba287a_0 jdcal 1.4 py36_0 jedi 0.13.2 py36_0 jeepney 0.4 py36_0 jinja2 2.10 py36_0 jpeg 9b h024ee3a_2 jsonschema 2.6.0 py36_0 jupyter 1.0.0 py36_7 jupyter_client 5.2.4 py36_0 jupyter_console 6.0.0 py36_0 jupyter_core 4.4.0 py36_0 jupyterlab 0.35.3 py36_0 jupyterlab_server 0.2.0 py36_0 keyring 17.1.1 py36_0 kiwisolver 1.0.1 py36hf484d3e_0 krb5 1.16.2 hbb41f41_0 conda-forge lazy-object-proxy 1.3.1 py36h14c3975_2 libarchive 3.3.3 h823be47_0 conda-forge libcurl 7.63.0 hbdb9355_0 conda-forge libedit 3.1.20170329 h6b74fdf_2 libffi 3.2.1 hd88cf55_4 libgcc-ng 8.2.0 hdf63c60_1 libgfortran-ng 7.3.0 hdf63c60_0 libiconv 1.15 h470a237_4 conda-forge liblief 0.9.0 h7725739_1 libpng 1.6.36 hbc83047_0 libsodium 1.0.16 h1bed415_0 libssh2 1.8.0 h5b517e9_3 conda-forge libstdcxx-ng 8.2.0 hdf63c60_1 libtiff 4.0.9 he85c1e1_2 libtool 2.4.6 h7b6447c_5 libuuid 1.0.3 h1bed415_2 libxcb 1.13 h1bed415_1 libxml2 2.9.8 h26e45fe_1 libxslt 1.1.32 h1312cb7_0 llvmlite 0.27.0 py36hd408876_0 locket 0.2.0 py36_1 lxml 4.3.0 py36hefd8a0e_0 lz4-c 1.8.1.2 h14c3975_0 lzo 2.10 h49e0be7_2 markupsafe 1.1.0 py36h7b6447c_0 matplotlib 3.0.2 py36h5429711_0 mccabe 0.6.1 py36_1 mistune 0.8.4 py36h7b6447c_0 mkl 2019.1 144 mkl-service 1.1.2 py36he904b0f_5 mkl_fft 1.0.10 py36ha843d7b_0 mkl_random 1.0.2 py36hd81dba3_0 more-itertools 5.0.0 py36_0 mpc 1.0.3 hec55b23_5 mpfr 3.1.5 h11a74b3_2 mpmath 1.1.0 py36_0 msgpack-python 0.5.6 py36h6bb024c_1 multipledispatch 0.6.0 py36_0 navigator-updater 0.2.1 py36_0 nbconvert 5.3.1 py36_0 nbformat 4.4.0 py36_0 ncurses 6.1 hfc679d8_2 conda-forge networkx 2.2 py36_1 nltk 3.4 py36_1 nose 1.3.7 py36_2 notebook 5.7.4 py36_0 numba 0.42.0 py36h962f231_0 numexpr 2.6.9 py36h9e4a6bb_0 numpy 1.15.4 py36h7e9f1db_0 numpy-base 1.15.4 py36hde5b4d6_0 numpydoc 0.8.0 py36_0 odo 0.5.1 py36_0 olefile 0.46 py36_0 openpyxl 2.5.12 py36_0 openssl 1.0.2p h14c3975_1002 conda-forge packaging 18.0 py36_0 pandas 0.23.4 py36h04863e7_0 pandoc 2.2.3.2 0 pandocfilters 1.4.2 py36_1 pango 1.42.4 h049681c_0 parso 0.3.1 py36_0 partd 0.3.9 py36_0 patchelf 0.9 he6710b0_3 path.py 11.5.0 py36_0 pathlib2 2.3.3 py36_0 patsy 0.5.1 py36_0 pcre 8.42 h439df22_0 pep8 1.7.1 py36_0 pexpect 4.6.0 py36_0 pickleshare 0.7.5 py36_0 pillow 5.4.1 py36h34e0f95_0 pip 18.1 py36_0 pixman 0.34.0 hceecf20_3 pkginfo 1.4.2 py36_1 pluggy 0.8.0 py36_0 ply 3.11 py36_0 prometheus_client 0.5.0 py36_0 prompt_toolkit 2.0.7 py36_0 psutil 5.4.8 py36h7b6447c_0 ptyprocess 0.6.0 py36_0 py 1.7.0 py36_0 py-lief 0.9.0 py36h7725739_1 pycodestyle 2.4.0 py36_0 pycosat 0.6.3 py36h14c3975_0 pycparser 2.19 py36_0 pycrypto 2.6.1 py36h14c3975_9 pycurl 7.43.0.2 py36hb7f436b_0 pyflakes 2.0.0 py36_0 pygments 2.3.1 py36_0 pylint 2.2.2 py36_0 pyodbc 4.0.25 py36he6710b0_0 pyopenssl 18.0.0 py36_0 pyparsing 2.3.0 py36_0 pyqt 5.9.2 py36h05f1152_2 pysocks 1.6.8 py36_0 pytables 3.4.4 py36h71ec239_0 pytest 4.0.2 py36_0 pytest-arraydiff 0.3 py36h39e3cac_0 pytest-astropy 0.5.0 py36_0 pytest-doctestplus 0.2.0 py36_0 pytest-openfiles 0.3.1 py36_0 pytest-remotedata 0.3.1 py36_0 python 3.6.6 h5001a0f_3 conda-forge python-dateutil 2.7.5 py36_0 python-libarchive-c 2.8 py36_6 pytz 2018.7 py36_0 pywavelets 1.0.1 py36hdd07704_0 pyyaml 3.13 py36h14c3975_0 pyzmq 17.1.2 py36h14c3975_0 qt 5.9.6 h8703b6f_2 qtawesome 0.5.3 py36_0 qtconsole 4.4.3 py36_0 qtpy 1.5.2 py36_0 readline 7.0 h7b6447c_5 requests 2.21.0 py36_0 rope 0.11.0 py36_0 ruamel_yaml 0.15.46 py36h14c3975_0 scikit-image 0.14.1 py36he6710b0_0 scikit-learn 0.20.2 py36hd81dba3_0 scipy 1.1.0 py36h7c811a0_2 seaborn 0.9.0 py36_0 secretstorage 3.1.0 py36_0 send2trash 1.5.0 py36_0 setuptools 40.6.3 py36_0 simplegeneric 0.8.1 py36_2 singledispatch 3.4.0.3 py36_0 sip 4.19.8 py36hf484d3e_0 six 1.12.0 py36_0 snappy 1.1.7 hbae5bb6_3 snowballstemmer 1.2.1 py36_0 sortedcollections 1.0.1 py36_0 sortedcontainers 2.1.0 py36_0 sphinx 1.8.2 py36_0 sphinxcontrib 1.0 py36_1 sphinxcontrib-websupport 1.1.0 py36_1 spyder 3.3.2 py36_0 spyder-kernels 0.3.0 py36_0 sqlalchemy 1.2.15 py36h7b6447c_0 sqlite 3.26.0 h7b6447c_0 statsmodels 0.9.0 py36h035aef0_0 swig 3.0.12 h38cdd7d_3 sympy 1.1 py36_0 tblib 1.3.2 py36_0 terminado 0.8.1 py36_1 testpath 0.4.2 py36_0 tk 8.6.8 hbc83047_0 toolz 0.9.0 py36_0 tornado 5.1.1 py36h7b6447c_0 tqdm 4.28.1 py36h28b3542_0 traitlets 4.3.2 py36_0 typed-ast 1.1.0 py36h14c3975_0 unicodecsv 0.14.1 py36_0 unixodbc 2.3.7 h14c3975_0 urllib3 1.24.1 py36_0 wcwidth 0.1.7 py36_0 webencodings 0.5.1 py36_1 werkzeug 0.14.1 py36_0 wheel 0.32.3 py36_0 widgetsnbextension 3.4.2 py36_0 wrapt 1.10.11 py36h14c3975_2 wurlitzer 1.0.2 py36_0 xlrd 1.2.0 py36_0 xlsxwriter 1.1.2 py36_0 xlwt 1.3.0 py36_0 xz 5.2.4 h14c3975_4 yaml 0.1.7 had09818_2 zeromq 4.2.5 hf484d3e_1 zict 0.1.3 py36_0 zlib 1.2.11 h7b6447c_3 zstd 1.3.7 h0b5b093_0
#3
Updated by Knödlseder Jürgen about 5 years ago
- Status changed from Resolved to Closed
- Assigned To set to Knödlseder Jürgen
- Target version set to 1.6.0
- % Done changed from 0 to 100
I added some text to the Known issues section of GammaLib and ctools.