Conda

The clas-unix team makes conda available on most of the linux systems in the /opt/anaconda3 directory. The conda command should already be in the search path of most users. Conda is a system for package management of multiple programming languages, but the most commonly used in CLAS is python. 

 

Conda Python environments

use the Cheat Sheet as a reference for commands.

Python environments act as containers that allow you to select a specific version of python and python modules to use for your project. Each environment can only have one version of python, and one version of each module installed. The conda environments are self contained and distinct from the system installed python. You can have multiple conda python environments, they just need to be installed in different directories.

The advantages of using conda environments:

  • The conda environments can be configured by regular users
  • The environment allows you to select a specific versions of python
  • Conda provides dependency resolution, so if you provide a list packages needed, it will attempt to solve for the prover version of python and other libraries that allow all of the packages requested to work.
  • The specifics of a configured conda environment can be exported as a yaml file that can be used to recreate the environment on a different server
  • conda environments have some limited management functionality that allows for showing changes and reverting most changes to the environment
  • The packages and modules in conda are built to work with each other in a consistent manner, but pip install is available as a last resort.

 

Verify Conda is available

(base) -bash-4.2$ conda --version
conda 4.9.2

Pre-requisite check

When using conda, it writes all of the packages you download to a cache in your home directory. If your home directory is on NFS, you will overrun your quota.

# user with an NFS home directory
df ~/ | grep nfs
clasnetappvm.iowa.uiowa.edu:/homedirs 2723960160 1979657856 744302304  73% /mnt/nfs/clasnetappvm/homedirs

# user with a local home directory won't have any output after running the command
df ~/ | grep nfs

If you do have an NFS home directory, you will need to create a local directory to  hold the package cache.

# assumes /data/not_backed_up exists, but use any local directory where there is space and you have write access
mkdir -p /data/not_backed_up/$USER/conda_pkgs
conda config --add pkgs_dirs "/data/not_backed_up/$USER/conda_pkgs"

 

All of the examples below assume that you have a local home directory. If you do not, set up your environments on local disk rather than in your home directory.

Create a Conda Environment

Identify a location that you have write access to to store the environment (in this case /home/ddholstad/conda_environments/) . We will create an environment that contains both nltk and numpy, letting conda select the newest version of python the these packages both work on. Conda will automatically download and install any packages that are prerequisites for the requested packages.

  • conda will create the directory
  • solve for dependencies
  • create a base python installation
  • install required packages

(base) -bash-4.2$ conda create -p /home/ddholstad/conda_environments/example1 nltk numpy
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/ddholstad/conda_environments/example1

  added / updated specs:
    - nltk
    - numpy

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    intel-openmp-2021.4.0      |    h06a4308_3561         4.2 MB
    mkl-2021.4.0               |     h06a4308_640       142.6 MB
    mkl-service-2.4.0          |   py39h7f8727e_0          59 KB
    mkl_fft-1.3.1              |   py39hd3c417c_0         182 KB
    mkl_random-1.2.2           |   py39h51133e4_0         309 KB
    numpy-1.21.2               |   py39h20f2e39_0          23 KB
    numpy-base-1.21.2          |   py39h79a1101_0         4.9 MB
    ------------------------------------------------------------
                                           Total:       152.2 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      pkgs/main/linux-64::_libgcc_mutex-0.1-main
  _openmp_mutex      pkgs/main/linux-64::_openmp_mutex-4.5-1_gnu
  blas               pkgs/main/linux-64::blas-1.0-mkl
  ca-certificates    pkgs/main/linux-64::ca-certificates-2021.10.26-h06a4308_2
  certifi            pkgs/main/linux-64::certifi-2021.10.8-py39h06a4308_0
  click              pkgs/main/noarch::click-8.0.3-pyhd3eb1b0_0
  intel-openmp       pkgs/main/linux-64::intel-openmp-2021.4.0-h06a4308_3561
  joblib             pkgs/main/noarch::joblib-1.1.0-pyhd3eb1b0_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.35.1-h7274673_9
  libffi             pkgs/main/linux-64::libffi-3.3-he6710b0_2
  libgcc-ng          pkgs/main/linux-64::libgcc-ng-9.3.0-h5101ec6_17
  libgomp            pkgs/main/linux-64::libgomp-9.3.0-h5101ec6_17
  libstdcxx-ng       pkgs/main/linux-64::libstdcxx-ng-9.3.0-hd4cf53a_17
  mkl                pkgs/main/linux-64::mkl-2021.4.0-h06a4308_640
  mkl-service        pkgs/main/linux-64::mkl-service-2.4.0-py39h7f8727e_0
  mkl_fft            pkgs/main/linux-64::mkl_fft-1.3.1-py39hd3c417c_0
  mkl_random         pkgs/main/linux-64::mkl_random-1.2.2-py39h51133e4_0
  ncurses            pkgs/main/linux-64::ncurses-6.3-h7f8727e_2
  nltk               pkgs/main/noarch::nltk-3.6.5-pyhd3eb1b0_0
  numpy              pkgs/main/linux-64::numpy-1.21.2-py39h20f2e39_0
  numpy-base         pkgs/main/linux-64::numpy-base-1.21.2-py39h79a1101_0
  openssl            pkgs/main/linux-64::openssl-1.1.1l-h7f8727e_0
  pip                pkgs/main/linux-64::pip-21.2.4-py39h06a4308_0
  python             pkgs/main/linux-64::python-3.9.7-h12debd9_1
  readline           pkgs/main/linux-64::readline-8.1-h27cfd23_0
  regex              pkgs/main/linux-64::regex-2021.8.3-py39h7f8727e_0
  setuptools         pkgs/main/linux-64::setuptools-58.0.4-py39h06a4308_0
  six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_0
  sqlite             pkgs/main/linux-64::sqlite-3.36.0-hc218d9a_0
  tk                 pkgs/main/linux-64::tk-8.6.11-h1ccaba5_0
  tqdm               pkgs/main/noarch::tqdm-4.62.3-pyhd3eb1b0_1
  tzdata             pkgs/main/noarch::tzdata-2021e-hda174b7_0
  wheel              pkgs/main/noarch::wheel-0.37.0-pyhd3eb1b0_1
  xz                 pkgs/main/linux-64::xz-5.2.5-h7b6447c_0
  zlib               pkgs/main/linux-64::zlib-1.2.11-h7f8727e_4

Proceed ([y]/n)? y

Downloading and Extracting Packages
mkl-service-2.4.0    | 59 KB     | ###################################################################################################################################### | 100%
intel-openmp-2021.4. | 4.2 MB    | ###################################################################################################################################### | 100%
numpy-1.21.2         | 23 KB     | ###################################################################################################################################### | 100%
mkl_fft-1.3.1        | 182 KB    | ###################################################################################################################################### | 100%
numpy-base-1.21.2    | 4.9 MB    | ###################################################################################################################################### | 100%
mkl-2021.4.0         | 142.6 MB  | ###################################################################################################################################### | 100%
mkl_random-1.2.2     | 309 KB    | ###################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate /home/ddholstad/conda_environments/example1
#
# To deactivate an active environment, use
#
#     $ conda deactivate

 

Activate the environment

(base) -bash-4.2$ conda activate /home/ddholstad/conda_environments/example1
(/home/ddholstad/conda_environments/example1) -bash-4.2$ which python
~/conda_environments/example1/bin/python
(/home/ddholstad/conda_environments/example1) -bash-4.2$ python
Python 3.9.7 (default, Sep 16 2021, 13:09:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk, numpy
>>>

 

Install additional packages as needed.

(/home/ddholstad/conda_environments/example1) -bash-4.2$ conda install pandas
Collecting package metadata (current_repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.11.0

Please update conda by running

    $ conda update -n base -c defaults conda

 

## Package Plan ##

  environment location: /home/ddholstad/conda_environments/example1

  added / updated specs:
    - pandas

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    bottleneck-1.3.2           |   py39hdd57654_1         125 KB
    numexpr-2.7.3              |   py39hb2eb853_0         183 KB
    pandas-1.3.4               |   py39h8c16a72_0         9.6 MB
    python-dateutil-2.8.2      |     pyhd3eb1b0_0         233 KB
    pytz-2021.3                |     pyhd3eb1b0_0         171 KB
    ------------------------------------------------------------
                                           Total:        10.3 MB

The following NEW packages will be INSTALLED:

  bottleneck         pkgs/main/linux-64::bottleneck-1.3.2-py39hdd57654_1
  numexpr            pkgs/main/linux-64::numexpr-2.7.3-py39hb2eb853_0
  pandas             pkgs/main/linux-64::pandas-1.3.4-py39h8c16a72_0
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.2-pyhd3eb1b0_0
  pytz               pkgs/main/noarch::pytz-2021.3-pyhd3eb1b0_0

Proceed ([y]/n)? y

Downloading and Extracting Packages
numexpr-2.7.3        | 183 KB    | ###################################################################################################################################### | 100%
pandas-1.3.4         | 9.6 MB    | ###################################################################################################################################### | 100%
bottleneck-1.3.2     | 125 KB    | ###################################################################################################################################### | 100%
python-dateutil-2.8. | 233 KB    | ###################################################################################################################################### | 100%
pytz-2021.3          | 171 KB    | ###################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Export A conda environment:

You can export your environment using the export command. Note use: "conda env export > envname.yml" to save as a file

/home/ddholstad/conda_environments/example1) -bash-4.2$ conda env export

name: /home/ddholstad/conda_environments/example1
channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=4.5=1_gnu
  - blas=1.0=mkl
  - bottleneck=1.3.2=py39hdd57654_1
  - ca-certificates=2021.10.26=h06a4308_2
  - certifi=2021.10.8=py39h06a4308_0
  - click=8.0.3=pyhd3eb1b0_0
  - intel-openmp=2021.4.0=h06a4308_3561
  - joblib=1.1.0=pyhd3eb1b0_0
  - ld_impl_linux-64=2.35.1=h7274673_9
  - libffi=3.3=he6710b0_2
  - libgcc-ng=9.3.0=h5101ec6_17
  - libgomp=9.3.0=h5101ec6_17
  - libstdcxx-ng=9.3.0=hd4cf53a_17
  - mkl=2021.4.0=h06a4308_640
  - mkl-service=2.4.0=py39h7f8727e_0
  - mkl_fft=1.3.1=py39hd3c417c_0
  - mkl_random=1.2.2=py39h51133e4_0
  - ncurses=6.3=h7f8727e_2
  - nltk=3.6.5=pyhd3eb1b0_0
  - numexpr=2.7.3=py39h22e1b3c_1
  - numpy=1.21.2=py39h20f2e39_0
  - numpy-base=1.21.2=py39h79a1101_0
  - openssl=1.1.1l=h7f8727e_0
  - pandas=1.3.4=py39h8c16a72_0
  - pip=21.2.4=py39h06a4308_0
  - python=3.9.7=h12debd9_1
  - python-dateutil=2.8.2=pyhd3eb1b0_0
  - pytz=2021.3=pyhd3eb1b0_0
  - readline=8.1=h27cfd23_0
  - regex=2021.8.3=py39h7f8727e_0
  - setuptools=58.0.4=py39h06a4308_0
  - six=1.16.0=pyhd3eb1b0_0
  - sqlite=3.36.0=hc218d9a_0
  - tk=8.6.11=h1ccaba5_0
  - tqdm=4.62.3=pyhd3eb1b0_1
  - tzdata=2021e=hda174b7_0
  - wheel=0.37.0=pyhd3eb1b0_1
  - xz=5.2.5=h7b6447c_0
  - zlib=1.2.11=h7f8727e_4
prefix: /home/ddholstad/conda_environments/example1

Remove an environment

(/home/ddholstad/conda_environments/example1) -bash-4.2$ conda deactivate
(base) -bash-4.2$ conda env remove -p /home/ddholstad/conda_environments/example1

Remove all packages in environment /home/ddholstad/conda_environments/example1:

(base) -bash-4.2$ ls -l /home/ddholstad/conda_environments/example1
ls: cannot access /home/ddholstad/conda_environments/example1: No such file or directory

create an environment from a yaml file

(base) -bash-4.2$ conda env create -f example1.yml -p /home/ddholstad/conda_environments/example1
Collecting package metadata (repodata.json): done
Solving environment: done

==> WARNING: A newer version of conda exists. <==
  current version: 4.9.2
  latest version: 4.11.0

Please update conda by running

    $ conda update -n base -c defaults conda

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate /home/ddholstad/conda_environments/example1
#
# To deactivate an active environment, use
#
#     $ conda deactivate