Using Anaconda Python on the Clusters

For researchers who have Python package requirements beyond the most common packages (Numpy, Scipy, Pandas), we recommend using Anaconda Python. Using conda environments you can manage which packages and even which versions of python you are using for different projects.

The Miniconda Module

For your convenience, we provide a relatively recent version of Miniconda (a base for Anaconda) as a module. It serves to bootstrap your personal environments. By using this module you do not need to download your own copy of Anaconda, which will prevent unnecessary file and storage usage in your home directory.

Setup Your Environment

Load the Miniconda module:

# Grace, Omega and Milgram
module load Langs/Python/miniconda

# Farnam & Ruddle
module load Python/miniconda

You can save this to your default module collection by using module save. See our module documentation for more details.

Create a conda environment:

Next, you will create a conda environment (saved by default to ~/.conda/envs) so that you can install any Python packages that you need.

Python 2

conda create -n my_project python=2.7

Python 3

conda create -n my_project python=3.6

You can replace my_project with any environment name that is meaningful to you and you can specify a different version of Python for an environment if necessary.

You might want multiple environments if you have packages that have conflicting prerequisites. If so, just rerun the previous commands to setup each environment as needed.

Activate & customize your environment:

You must first load your conda environment before installing or using the packages in it. You only need to install packages once per environment.

source activate my_project

# install packages, e.g. numpy
conda install numpy
# or, if not available in conda, e.g. ansible
pip install ansible

Using Your Environment

Interactively:

Your conda environments will not follow you into job allocations, so make sure to activate them after your interactive job begins. To re-enter your environment at any time, you just need the following:

source activate my_project

In a job submission script:

To make sure that you are running in your project environment in a submission script, make sure to include the following lines in your submission script before running any other Python commands or scripts (but after your Slurm directives):

# Grace, Omega and Milgram
module load Langs/Python/miniconda

# Farnam & Ruddle
module load Python/miniconda

source activate my_project

python #MY PYTHON SCRIPT HERE

Troubleshooting

"Permission Denied"

If you get a permission denied error while trying to conda or pip install a package, make sure you have created an environment or activated an existing one first.

"-bash: activate: No such file or directory"

If you get the above error, it is likely that you don't have the necessary module file loaded. Try loading the appropriate module and rerunning your source activate my_project command.

"could not find environment:"

This error means that the version of Anaconda/Miniconda you have loaded doesn't recognize the environment name you have supplied. Make sure you have the Miniconda module loaded (and not a different Python module) and have previously created this environment. You can see a list of previously created environments by running:

conda info --envs

Additional Conda Commands

List installed packages:

source activate my_project
conda list

Delete a conda environment:

conda env remove --name my_project

Sharing your conda environment:

If you want to share or back up a conda environment, you can export it to a file. To do so you need to run the following, replacing my_project with the desired environment.

source activate my_project
conda env export > my_project_environment.yml
# on another machine or account, run
conda env create -f my_project_environment.yml

Installing to a different directory:

conda will look in the directory/directories specified in the environment variable CONDA_ENVS_PATH for places to find and install environments to. If you want your environments stored in ~/project/conda_envs, for example, you would need to set the variable something like this.

echo "export CONDA_ENVS_PATH=~/project/conda_envs:$CONDA_ENVS_PATH" >> ~/.bashrc
source ~/.bashrc

Conda documentation:

Please see the excellent conda documentation on the official Anaconda website for some additional troubleshooting information and features provided by conda.