Page tree
Skip to end of metadata
Go to start of metadata
This page:


One of the benefits of working with a container is that it can simplify software installation and deployment. With our HPC systems we recommend the use of Singularity due to HPC-friendly usage, thus also extend the recommendation for usage on Nimbus. If you are familiar with Docker, you may also use existing Docker images with Singularity.

Singularity is pre-installed on all Nimbus images with the "- Pawsey" suffix. If you have selected such image for your instance, you can skip step 1. Click here for more information on Nimbus images.

The workflow for Singularity is as follows:

  1. Download and install Singularity 
  2. Build a Singularity container by:
    1. Pulling an existing container from Docker Hub or Container Library, 
    2. Building a Singularity container with a Singularity definition file, or
    3. Assembling a Singularity container from an external container
  3. Run the image using Singularity



Install Singularity on Nimbus (v 3.5.2) with the following instructions:

Install system dependencies:

>sudo apt-get update && \
  sudo apt-get install -y build-essential \
  libseccomp-dev pkg-config squashfs-tools cryptsetup

Install latest Golang (v1.14.2 at time of writing):

>export VERSION=1.14.2 OS=linux ARCH=amd64
>wget -O /tmp/go${VERSION}.${OS}-${ARCH}.tar.gz${VERSION}.${OS}-${ARCH}.tar.gz && \
  sudo tar -C /usr/local -xzf /tmp/go${VERSION}.${OS}-${ARCH}.tar.gz

Set up the environment for Go:

>echo 'export GOPATH=${HOME}/go' >> ~/.bashrc &&   echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc &&   source ~/.bashrc

Install golangci-lint:

>curl -sfL |   sh -s -- -b $(go env GOPATH)/bin v1.15.0

Download the Singularity version 3.5.2 release:

>mkdir -p ${GOPATH}/src/ && \
  cd ${GOPATH}/src/ && \
  git clone && \
  cd singularity
>git checkout v3.5.2

Then compile it:

>cd ${GOPATH}/src/ && \
  ./mconfig && \
  cd ./builddir && \
  make && \
  sudo make install

Check your installation:

> singularity version

You may like to add the following line to your ~/.bashrc file for bash completions to work with Singularity commands every time you open a new shell:

>. /usr/local/etc/bash_completion.d/singularity

Pull an existing image

The following commands will enable pulling of existing images from various container repositories, as well as building from existing containers or building from a definition file. For a more in-depth tutorial on how to perform these commands, you can visit our tutorial page here: or the Singularity user guide here:

The default directory location for the Singularity image cache is $HOME/.singularity/cache. This can be changed to your /data directory, as your /home directory would usually have a much smaller volume. You can redefine the path to the cache dir by setting the variable SINGULARITY_CACHEDIR. E.g.:

>sudo mkdir /data/singularity_cache
>export SINGULARITY_CACHEDIR=/data/singularity_cache

To import Docker images from e.g. Docker Hub, you can use the singularity pull command. As Docker images are written in layers, Singularity pulls the layers instead of just downloading the image, then combines them into a Singularity SIF format container.

>sudo mkdir sif_lib
>sudo singularity pull --dir sif_lib docker://library/image:tag
  • The --dir flag specifies the image to be downloaded to a location, e.g. sif_lib in this case
  • docker:// indicates that you're pulling from the Docker Hub library 
  • library is the hub user 
  • image is the image or repository name you're pulling
  • tag is the Docker Hub tag that identifies which image to pull

Build a Singularity container

To build a Singularity container with a Singularity definition file, the singularity build command is also used. You would execute the following command in the directory where you want to store the container.

>sudo singularity build lolcow.sif lolcow.def
  • The .sif file is the container you are building
  • The .def file is the Singularity definition file that is required to build the image. This has to be in the same directory as the build.

More information on Singularity definition files can be found here: Singularity Definition Files

Assemble a Singularity container

If you require to modify an existing container, Singularity allows changes to be made after pulling the image, before building it into a Singularity container. However, this method may result in inability to track changes that you have made to a container, as you will be unable to read the definition file. Pawsey recommends that you build and modify your containers using a Singularity definition file.

Only use this method if absolutely necessary, as you won't be able to track the changes you have made, i.e. there is no definition file recipe to look back on

To assemble a Singularity container from an external container, the singularity build command is used with the --sandbox flag. With this method, layers of the external container are first downloaded, then built in your designated directory.

>sudo singularity build --sandbox lolcow/ library://sylabs-jms/testing/lolcow
  • The --sandbox flag allows the container to be created with a writable directory for making changes to
  • The lolcow/ directory is where you are the building the container
  • library:// indicates that you are pulling from the Sylabs Container Library
  • syllabus-jms is the author's library, testing is the repository, and lolcow is the image name

To then make changes to the container, you would use the --writable flag as follows:

>sudo singularity shell --writable lolcow/

Once ready to covert the sandbox container to a SIF container for execution, you will then run the following:

>sudo singularity build lolcow.sif lolcow/

Working with Images

To see all of the images in your local repository on Pawsey you can use the singularity cache list command:

>sudo singularity cache list

NAME                     DATE CREATED           SIZE             TYPE
ubuntu_latest.sif        2019-10-21 13:19:50    28.11 MB         library
ubuntu_18.04.sif         2019-10-21 13:19:04    37.10 MB         library
ubuntu_18.04.sif         2019-10-21 13:19:40    25.89 MB         oci

There 3 containers using: 91.10 MB, 6 oci blob file(s) using 26.73 MB of space.
Total space used: 117.83 MB

If you are running out of disk space, you can clean it up, e.g. to wipe everything use the -a flag (use -f instead from Singularity version 3.4 on):

>sudo singularity cache clean -a

Running Singularity

To run Singularity, there are two main commands that you will use.

>sudo singularity run sif_lib/your-container.sif 
  • This command assumes that your container has a runscript (or run functions) that will be executed on running the container as above.

>sudo singularity exec sif_lib/your-container.sif <command>
  • This command allows you to execute a specific command you want to container to run. Replace <command> accordingly.

Bind mounting host directories

With Singularity you are able to mount your host directory to the container. For example, you may want to mount a directory where you would like to read and write files in from the container. The syntax required to do so is:

>sudo singularity exec -B /path/to/host/directory:/path/in/container sif_lib/your-container.sif <command>
  • singularity exec allows you to execute the container with a specific command that is placed at the end of the string
  • -B is the flag for bind mounting the directory to the container
  • Note: you can also remove ":/path/in/container" if you just need to bind a path/directory to the main directory of the container.

Containers with Python and R

A note for Singularity containers that have Python or R built in, the flag -C is required to run the container in an isolated mode. If you require reading and/or writing from a local directory, you may use the -C flag in conjunction with the -B flag:

>sudo singularity exec -C sif_lib/rstudio_latest.sif R
>sudo singularity exec -B /path/to/host/directory:/path/in/container -C sif_lib/rstudio_latest.sif
  • sif_lib is the directory you pulled your container to.
  • rstudio_latest.sif is a container that was pulled as such: sudo singularity pull docker://rocker/rstudio
  • R is the command that you are asking the container to execute, which in this case opens up R from within the container.
  • Note: you can also remove ":/path/in/container" if you just need to bind a path/directory to the main directory of the container.

Using GPUs

Singularity allows users to make use of GPUs within their containers, by adding the runtime flag --nv. Here's an example of running Gromacs, a popular molecular dynamics package, among the ones that have been optimised to run on GPUs through Nvidia containers:

export SINGULARITY_CACHEDIR=/data/singularity_cache

# Run Gromacs preliminary step with container
>sudo singularity exec --nv $SIFPATH/gromacs_2018.2.sif \
    					   gmx grompp -f pme.mdp

# Run Gromacs MD with container
>sudo singularity exec --nv $SIFPATH/gromacs_2018.2.sif \
    					   gmx mdrun -ntmpi 1 -nb gpu -pin on -v \
                           -noconfout -nsteps 5000 -s topol.tpr -ntomp 1

Singularity Build Tips & Tricks

In general, here are some recommended best practices:

  • Minimize image size
    • Each %post and %files command in a Singularity definition file generates another layer in the container, increasing its size
    • To minimize image size, use multi-line commands, and clean up package manager caches:

      	apt-get update \
      	&& apt-get install -y \
             autoconf \
             automake \
             gcc \
             g++ \
             python \
             python-dev \
          && apt-get clean all \
          && rm -rf /var/lib/apt/lists/*
  • Software bloat

    • Only install the software you need for a given application into a container
  • Modularity
    • Creating giant, monolithic containers with every possible application you could need is bad practice.  It increases image size, reduces performance, and increases complexity.  Containers should only contain a few (ideally only 1) applications that you'll use. You can chain together workflows that use multiple containers, meaning if you need to change a particular portion you only need to update a single, small container.

Additional Documentation

In this section:

  • No labels