For researchers who have Python package requirements beyond the most common packages (Numpy, Scipy, Pandas), we recommend using Anaconda Python. Using conda environments you can manage which packages and even which versions of python you are using for different projects.
The Miniconda Module
For your convenience, we provide a relatively recent version of Miniconda (a base for Anaconda) as a module. It serves to bootstrap your personal environments. By using this module you do not need to download your own copy of Anaconda, which will prevent unnecessary file and storage usage in your home directory.
Note: If you are on Milgram and run out of space in your home directory for Python, you can either reinstall your environment in your project space (see below) or contact us at email@example.com for help with your home quota.
Setup Your Environment
Load the Miniconda module:
# Grace, Omega and Milgram module load Langs/Python/miniconda # Farnam & Ruddle module load Python/miniconda
You can save this to your default module collection by using
module save. See our module documentation for more details.
Next, you will create a conda environment (saved by default to
~/.conda/envs) so that you can install any Python packages that you need.
Installing to a different directory:
conda will look in the directory/directories specified in the environment variable
CONDA_ENVS_PATH for places to find and install environments. If you want your environments stored in a directory where your quotas are higher, for example,
~/project/conda_envs, you would need to set this variable to something like.
echo "export CONDA_ENVS_PATH=~/project/conda_envs:$CONDA_ENVS_PATH" >> ~/.bashrc source ~/.bashrc
conda create -n my_project python=2.7
conda create -n my_project python=3.6
You can replace
my_project with any environment name that is meaningful to you and you can specify a different version of Python for an environment if necessary.
You might want multiple environments if you have packages that have conflicting prerequisites. If so, just rerun the previous commands to setup each environment as needed.
Activate & customize your environment:
You must first load your conda environment before installing or using the packages in it. You only need to install packages once per environment.
source activate my_project # install packages, e.g. numpy conda install numpy # or, if not available in conda, e.g. ansible pip install ansible
Using Your Environment
Your conda environments will not follow you into job allocations, so make sure to activate them after your interactive job begins. To re-enter your environment at any time, you just need the following:
source activate my_project
In a job submission script:
To make sure that you are running in your project environment in a submission script, make sure to include the following lines in your submission script before running any other Python commands or scripts (but after your Slurm directives):
# Grace, Omega and Milgram module load Langs/Python/miniconda # Farnam & Ruddle module load Python/miniconda source activate my_project python #MY PYTHON SCRIPT HERE
If you get a permission denied error while trying to conda or pip install a package, make sure you have created an environment or activated an existing one first.
"-bash: activate: No such file or directory"
If you get the above error, it is likely that you don't have the necessary module file loaded. Try loading the appropriate module and rerunning your
source activate my_project command.
"could not find environment:"
This error means that the version of Anaconda/Miniconda you have loaded doesn't recognize the environment name you have supplied. Make sure you have the Miniconda module loaded (and not a different Python module) and have previously created this environment. You can see a list of previously created environments by running:
conda info --envs
Additional Conda Commands
List installed packages:
source activate my_project conda list
Delete a conda environment:
conda env remove --name my_project
Sharing your conda environment:
If you want to share or back up a conda environment, you can export it to a file. To do so you need to run the following, replacing
my_project with the desired environment.
source activate my_project conda env export > my_project_environment.yml # on another machine or account, run conda env create -f my_project_environment.yml
Please see the excellent conda documentation on the official Anaconda website for some additional troubleshooting information and features provided by conda.