Bug #3628

ctools and gammalib environment not correctly initialised when conda is running in tcsh

Added by Knödlseder Jürgen almost 3 years ago. Updated almost 3 years ago.

Status:ClosedStart date:04/28/2021
Priority:ImmediateDue date:
Assigned To:Knödlseder Jürgen% Done:

100%

Category:-
Target version:1.7.4
Duration:

Description

Martin Giard had a problem when trying to install and unit test ctools, the issue being that the CTOOLS environment variable was not set. It turned out that he was running anaconda under tcsh, and that the activation scripts are only provided in shell.

See https://github.com/conda/conda-build/issues/3880 for a related issue. Possibly adding specific csh activation scripts can solve the problem.

Recurrence

No recurrence.

History

#1 Updated by Knödlseder Jürgen almost 3 years ago

  • % Done changed from 0 to 10

In order to try reproducing the problem on my Mac I installed a new miniconda version under tcsh:

$ tcsh
% ~/Downloads/Miniconda2-latest-MacOSX-x86_64.sh
% conda env list
# conda environments:
#
                         /Users/jurgen/anaconda3
                         /Users/jurgen/anaconda3/envs/astropy
                         /Users/jurgen/anaconda3/envs/ctools-1.6.1
                         /Users/jurgen/anaconda3/envs/ctools-1.7.3
                         /Users/jurgen/anaconda3/envs/ctools-paper
                         /Users/jurgen/anaconda3/envs/rootenv
                         /Users/jurgen/anaconda3/envs/sharepoint
base                  *  /Users/jurgen/miniconda2-tcsh
% conda info

     active environment : None
       user config file : /Users/jurgen/.condarc
 populated config files : /Users/jurgen/.condarc
          conda version : 4.8.3
    conda-build version : not installed
         python version : 2.7.18.final.0
       virtual packages : __osx=10.14.6
       base environment : /Users/jurgen/miniconda2-tcsh  (writable)
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://conda.anaconda.org/cta-observatory/osx-64
                          https://conda.anaconda.org/cta-observatory/noarch
                          https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/jurgen/miniconda2-tcsh/pkgs
                          /Users/jurgen/.conda/pkgs
       envs directories : /Users/jurgen/miniconda2-tcsh/envs
                          /Users/jurgen/.conda/envs
               platform : osx-64
             user-agent : conda/4.8.3 requests/2.22.0 CPython/2.7.18 Darwin/18.7.0 OSX/10.14.6
                UID:GID : 501:20
             netrc file : None
           offline mode : False

I then followed the gammalib installation instructions:
% conda update -n base -c defaults conda
% conda create -n tcsh python=2.7
% conda activate tcsh
CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.
% conda activate tcsh
modified      /Users/jurgen/.tcshrc
==> For changes to take effect, close and re-open your current shell. <==
% exit
exit
macp0135:~ jurgen$ tcsh
% conda activate tcsh
% conda install gammalib
% conda activate tcsh
% python -c 'import gammalib; gammalib.test()'
% conda install ctools
% python -c 'import ctools; ctools.test()'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/local/gamma/lib/python2.7/site-packages/ctools/tools.py", line 2060, in test
    test_python_ctools.test(installed=True)
  File "/usr/local/gamma/lib/python2.7/site-packages/ctools/tests/test_python_ctools.py", line 102, in test
    os.system('cp -r %s/syspfiles/*.par pfiles/' % (os.environ['CTOOLS']))
  File "/Users/jurgen/miniconda2-tcsh/envs/tcsh/lib/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'CTOOLS'
While GammaLib seems to work, ctools doesn’t. This reproduces Martin’s problem.

#2 Updated by Knödlseder Jürgen almost 3 years ago

  • Status changed from New to In Progress
  • % Done changed from 10 to 20

I added the file 01-ctools-activate.csh to the /Users/jurgen/miniconda2-tcsh/envs/tcsh/etc/conda/activate.d directory:

#!/bin/csh

setenv CTOOLS ${CONDA_PREFIX}
source ${CTOOLS}/bin/ctools-init.csh
unsetenv PYTHONPATH
unsetenv LD_LIBRARY_PATH
unsetenv DYLD_LIBRARY_PATH

Doing now
% conda activate tcsh
(tcsh) [macp0135:~] jurgen% echo $CTOOLS
/Users/jurgen/miniconda2-tcsh/envs/tcsh
(tcsh) [macp0135:~] jurgen% python -c 'import ctools; ctools.test()'
(tcsh) [macp0135:~] jurgen% python -c 'import cscripts; cscripts.test()'
(tcsh) [macp0135:~] jurgen% python -c 'import gammalib; gammalib.test()'
works!

#3 Updated by Knödlseder Jürgen almost 3 years ago

  • % Done changed from 20 to 30

I added 00-gammalib-activate.csh and 01-gammalib-deactivate.csh to GammaLib’s dev directory and typed in the GammaLib directory the following to build GammaLib locally:

$ /Users/jurgen/anaconda3/bin/conda-build dev/conda.recipe
BUILD START: ['gammalib-1.7.4-py35_1.tar.bz2']
Could not download http://cta.irap.omp.eu/ctools/releases/gammalib/gammalib-1.7.4.tar.gz
This means that code downloading is needed to make this work.

#4 Updated by Knödlseder Jürgen almost 3 years ago

  • % Done changed from 30 to 40

Following the bugfix 1.7.4 release that includes conda activation scripts for csh I tested the gammalib installation using conda under tcsh on Mac OS:

% conda create -n tcsh python=2.7
% conda activate tcsh
% conda install gammalib
The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    gammalib-1.7.4             |   py27h59582e6_1        26.7 MB  cta-observatory
    ------------------------------------------------------------
                                           Total:        26.7 MB
% echo $GAMMALIB
GAMMALIB: Undefined variable.
% conda activate tcsh
% echo $GAMMALIB
GAMMALIB: Undefined variable.
% ls /Users/jurgen/miniconda2-tcsh/envs/tcsh/etc/conda/activate.d/
00-gammalib-activate.sh
The csh activation script is missing, as well as the deactivation script. I forgot to add the inclusion of these scripts in the tarball.

#5 Updated by Knödlseder Jürgen almost 3 years ago

  • % Done changed from 40 to 50

After adding the csh scripts to the conda distribution I redid the check. Now everything works nicely. It turns out that the re-activation after installation of GammaLib is not needed anymore, conda seems to do the reactivation automatically.

% echo $GAMMALIB
GAMMALIB: Undefined variable.
% conda create -n tcsh python=2.7
% conda activate tcsh
% echo $GAMMALIB
GAMMALIB: Undefined variable.
% conda install gammalib
The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    gammalib-1.7.4             |   py27h59582e6_1        26.7 MB  cta-observatory
    ------------------------------------------------------------
                                           Total:        26.7 MB
% echo $GAMMALIB
/Users/jurgen/miniconda2-tcsh/envs/tcsh
% python -c 'import gammalib; gammalib.test()'

#6 Updated by Knödlseder Jürgen almost 3 years ago

I verified the same procedure with Python 3.5, 3.6, 3.7 and 3.8 and they all work as for Python 2.7.

I also verified that using bash no re-activation of the environment is needed. This seems therefore a generic feature of conda now:

echo $GAMMALIB

$ conda create -n bash python=2.7
$ conda activate bash
$ echo $GAMMALIB

$ conda install gammalib
$ echo $GAMMALIB
/Users/jurgen/anaconda3/envs/bash
$ python -c 'import gammalib; gammalib.test()'
$ conda deactivate
$ echo $GAMMALIB
Note that deactivate properly removes the environment variable. Same was also checked for tcsh.

#7 Updated by Knödlseder Jürgen almost 3 years ago

  • % Done changed from 50 to 60

I now did the same thing for ctools, first on tcsh. Everything works as expected.

echo $CTOOLS
CTOOLS: Undefined variable.
% conda create -n tcsh python=2.7
% conda activate tcsh
% echo $CTOOLS
CTOOLS: Undefined variable.
% conda install ctools
The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ctools-1.7.4               |   py27h97731cd_1         7.5 MB  cta-observatory
    ------------------------------------------------------------
                                           Total:         7.5 MB
% echo $CTOOLS
/Users/jurgen/miniconda2-tcsh/envs/tcsh
% python -c 'import ctools; ctools.test()'
% python -c 'import cscripts; cscripts.test()'
% python -c 'import gammalib; gammalib.test()'
% conda deactivate
% echo $CTOOLS
CTOOLS: Undefined variable.

#8 Updated by Knödlseder Jürgen almost 3 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 60 to 100

It also works perfectly for Python 3.5 - 3.8.

I also checked for bash, and also here everything is fine. Close the issue now.

Also available in: Atom PDF