Page tree
Skip to end of metadata
Go to start of metadata

Pawsey provides and maintains a number of prebuilt libraries and applications on Setonix, most of which are installed and managed through Spack. This page outlines how users can use Spack to install additional software or different builds of existing software that are not provided by Pawsey supported modules.  

On this page

Before you begin

Read Software Stack Policies before proceeding with any local installations.

Before using Spack to install a package, run the following command to check whether it is already available:

module avail <package-name>

To familiarise yourself with Spack features and concepts, read the Spack documentation (external site).

Motivation for using Spack

Spack provides a simple interface for installing packages. Amongst other benefits, Spack provides human-readable installation commands, is able to automatically build the dependency tree for a large number of packages, and allows different installation configurations to be selected. As a result, Spack lessens the need to know the variety of installation procedures in use, such as automake and cmake

Spack at Pawsey

Spack has been configured by Pawsey staff to work out of the box for most usage scenarios. In particular, the following aspects have been tuned:

  • Usage of available compilers
  • Usage of available performance libraries
  • Awareness of system-wide packages
  • Directory trees for installed packages and modules
  • Naming convention and features of generated modulefiles

Spack is used at Pawsey to deploy a significant fraction of the scientific software stack including HPC applications and HPC libraries, as well as some compilers and utilities.

Installing Python and R packages

The Spack setup at Pawsey is capable of handling both system-wide and user package installations. However, the current implementation does not properly generate user modulefiles for interpreted languages such as Python and R.

For Python package installations, we recommend using pip or setuptools, see: Installing Python Packages

For R package installations, we recommend using the R package manager, see: Installing R Packages

Installing GPU applications

The 0.17.0 version of Spack currently deployed on Setonix does not support installations of applications depending on ROCm, the AMD GPU computing platform. We will soon deploy a new software stack with a more recent version of Spack.

Using Spack

First, the Spack module has to be loaded:

    $ module load spack/0.17.0

Users can invoke Spack with the spack command. spack accepts a subcommand, or Spack command, as the first argument:

    $ spack <subcommand>

Use the following subcommands in order:

  1. Use list to look for package recipes.
  2. Use info to inspect available options for a given package.
  3. Use spec to test installation and required dependencies for a given package.

    Always test an installation with spec before running install.

  4. Finally, proceed with the install subcommand for a given package.

A comprehensive list of Spack commands can be inspected by using variations of the help subcommand.

$ spack help
$ spack help --all
$ spack <subcommand> -h

Solving the "Disk Quota Exceeded' error

All spack install ...  commands, and any other command generating files on the /software filesystem, must be executed using the sg Linux utility to make sure the process is run with the correct group ID. All files under /software must have the Linux group of the project they are created by, to contribute towards the correct quota limit. If you fail to do so, these files are accounted for in your personal quote that is meant to limit /home  usage and is much more limited. The syntax is the following:

sg <yourprojectcode> -c 'spack install <spec>'.

For instance, Pawsey staff may install the namd scientific software using the command

sg pawsey0001 -c 'spack install namd'.

Example installation process with NAMD

Suppose we want to install the NAMD software for molecular dynamics simulations. First, let's search Spack for an appropriate package.

Terminal 1: spack list command
$ spack list namd
==> 1 packages.

Spack has found one package called namd. It sounds like it's the one we're looking for.

The namd package in Spack has several build time options and dependencies. These options can be explored by using the info subcommand. 

Terminal 2: spack info command
$ spack info namd
MakefilePackage:   namd

    NAMD is a parallel molecular dynamics code designed for high-performance
    simulation of large biomolecular systems.



Preferred version:
    2.14      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.14_Source.tar.gz

Safe versions:
    master    [git] on branch master
    2.15a1    [git] at tag release-2-15-alpha-1
    2.14      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.14_Source.tar.gz
    2.13      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.13_Source.tar.gz
    2.12      file:///group/pawsey0001/mulan/spack-builds/spack/bin/NAMD_2.12_Source.tar.gz

    Name [Default]      Allowed values          Description
    ================    ====================    ============================================

    cuda [off]          on, off                 Build with CUDA
    cuda_arch [none]    none, 52, 12, 61,       CUDA architecture
                        37, 60, 30, 11, 10,
                        20, 86, 35, 72, 62,
                        70, 13, 53, 32, 50,
                        75, 80, 21
    fftw [3]            none, 2, 3, mkl,        Enable the use of FFTW/FFTW3/MKL FFT/AMDFFTW
    interface [none]    none, tcl, python       Enables TCL and/or python interface

Installation Phases:
    edit    build    install

Build Dependencies:
    amdfftw  charmpp  cuda  fftw  intel-mkl  python  tcl

Link Dependencies:
    amdfftw  charmpp  cuda  fftw  intel-mkl  python  tcl

Run Dependencies:

Virtual Packages:

The info output lists the build time options under Variants, along with the build and link dependencies of the package. 

The dependencies required can be explored by running the spec subcommand:

Terminal 3: spack spec command
$ spack spec namd # default installation of namd and the libraries it depends on 
Input spec

namd@2.14%gcc@11.2.0~cuda cuda_arch=none fftw=3 interface=none patches=cdcbc3846be1dfd6dbf958177c703e15ef5343e1461ed68d02be8fd0512429d5 arch=linux-sles15-zen3
    ^charmpp@6.10.2%gcc@11.2.0~cuda~omp~papi+production~pthreads+shared+smp~syncft~tcp~tracing backend=netlrts build-target=LIBS pmi=none arch=linux-sles15-zen3
    ^fftw@ precision=double,float arch=linux-sles15-zen3

Spack will produce the default installation, listing for each package that needs to be installed the full compilation options. Focusing on namd, Spack reports the following information:

  • The package and version (namd@2.14)
  • The compiler and version (%gcc@11.2.0)
  • Information related to the variants and the architecture that the package will be built for (in this example, the linux-sles15 operating system using the zen3 CPU architecture)

Variants are listed with the following specifiers:

  • +  indicates that the compilation option is enabled
  • ~ (no prepending space) or - (with prepending space) indicate that the compilation option is disabled
  • Variants with multiple values are listed as option_name=value 

In this case, when namd is compiled cuda is disabled, fftw is set to 3 (use fftw3 library) and no interface has been used. 

Dependencies are preceded by a ^. For instance, here namd  depends on charmpp@6.10.2 and fftw@, with each dependency listing which compiler and variants it will be built with. 

Explore changes in the build process with spec by adding the variants to the package name.

Terminal 4: spack spec with variants
$ spack spec namd +cuda fftw=mkl interface=python #setting a specific variant 
Input spec
namd+cuda fftw=mkl interface=python

namd@2.14%gcc@11.2.0+cuda cuda_arch=none fftw=mkl interface=python patches=cdcbc3846be1dfd6dbf958177c703e15ef5343e1461ed68d02be8fd0512429d5 arch=linux-sles15-zen3
    ^charmpp@6.10.2%gcc@11.2.0~cuda~omp~papi+production~pthreads+shared+smp~syncft~tcp~tracing backend=netlrts build-target=LIBS pmi=none arch=linux-sles15-zen3
    ^cuda@11.3.0%gcc@11.2.0~dev arch=linux-sles15-zen3
        ^libxml2@2.9.10%gcc@11.2.0~python arch=linux-sles15-zen3
            ^libiconv@1.16%gcc@11.2.0 arch=linux-sles15-zen3
            ^pkgconf@1.4.2%gcc@11.2.0 arch=linux-sles15-zen3
            ^xz@5.2.4%gcc@11.2.0~pic libs=shared,static arch=linux-sles15-zen3
            ^zlib@1.2.11%gcc@11.2.0+optimize+pic+shared arch=linux-sles15-zen3
    ^intel-mkl@2020.4.304%gcc@11.2.0~ilp64+shared threads=none arch=linux-sles15-zen3
        ^cpio@2.12%gcc@11.2.0 patches=d22633c368b8aedf4c08b23b6fbaa81a52404c8943ab04926404083ac10f1a4b arch=linux-sles15-zen3
    ^python@3.6.13%gcc@11.2.0+bz2+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib arch=linux-sles15-zen3
    ^tcl@8.6.11%gcc@11.2.0 arch=linux-sles15-zen3

Finally, install the package.

The spack command shown in terminal 5 will try to install and build all of the required packages in the user location configured by Pawsey, which is:


In this example:

  • <arch> = linux-sles15-zen3
  • <compiler> = gcc-11.2.0
  • <package>-<version>-<unique_hash> = namd-2.12-x3jinxvmanf5dm6zujstnpalmlzi4hy5/
Terminal 5: spack install with variants and dependencies
$ # Before install, check the spec using specific version of fftw
$ spack spec namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches ^mpich@3.4.1
Input spec
namd@2.12~cuda fftw=3 interface=python

namd@2.12%gcc@11.2.0~cuda cuda_arch=none fftw=3 interface=python arch=linux-sles15-zen3
    ^charmpp@6.7.1%gcc@11.2.0~cuda~omp~papi+production~pthreads+shared+smp~syncft~tcp~tracing backend=netlrts build-target=LIBS patches=92d507d8b72ac01f114f7772510049cc7656f93daeadef46e1f2b00ad8533d2b,e636c3f88a6944c87d910d4c8fdf25146b60f57e09228497a6cc01c7dbae9569 pmi=none arch=linux-sles15-zen3
    ^fftw@3.3.4%gcc@11.2.0+mpi+openmp+pfft_patches patches=4c813199aa39c0d2b23259ee3c3a58c9c96a3c0d09fe3485a33f299b95e0d9e7 precision=double,float arch=linux-sles15-zen3
        ^autoconf@2.69%gcc@11.2.0 arch=linux-sles15-zen3
        ^automake@1.15.1%gcc@11.2.0 arch=linux-sles15-zen3
        ^libtool@2.4.6%gcc@11.2.0 arch=linux-sles15-zen3
        ^mpich@3.4.1%gcc@11.2.0~argobots+fortran+hwloc+hydra+libxml2+pci+romio+slurm~verbs+wrapperrpath device=ch4 netmod=ofi pmi=pmi arch=linux-sles15-zen3
    ^python@3.6.13%gcc@11.2.0+bz2+ctypes+dbm~debug+libxml2+lzma+nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib arch=linux-sles15-zen3
    ^tcl@8.6.11%gcc@11.2.0 arch=linux-sles15-zen3
        ^zlib@1.2.11%gcc@11.2.0+optimize+pic+shared arch=linux-sles15-zen3
$ # the spec looks okay, let's install it
$ sg <projectcode> -c 'spack install namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches'

Spack would normally download the source code for any packages that it needs to install. The source code for NAMD has a particular restriction that it cannot be downloaded by spack but must be obtained separately:

Terminal 6: namd install issue
$ sg <projectcode> -c 'spack install namd@2.12 ~cuda fftw=3 interface=python ^fftw@3.3.4 +mpi +openmp +pfft_patches'
==> Installing namd-2.12-tey6bdfijoegx3dkxfc553nckfecmhku
==> No binary for namd-2.12-tey6bdfijoegx3dkxfc553nckfecmhku found: installing from source
==> Error: FetchError: Manual download is required for namd. Refer to for download instructions.

This can be addressed by first manually downloading the source tarball into your current directory where spack will detect it.

Install a package for the entire project team

Pawsey provides a utility script, called, to perform package installations with Spack that need to be visible to the entire project team, rather than just a single user. 

This script is a drop-in replacement for the spack command:

$ spec <arguments>
$ install <arguments>

The key differences compared to spack are the locations where packages are installed and modulefiles are created:

  • Software: /software/projects/<project-id>/setonix/software/<arch>/<compiler>/<package>-<version>-<unique_hash>
  • Modulefiles: /software/projects/<project-id>/setonix/modules/<cpu-arch>/<compiler-name>/<compiler-version>/<package>/<version>-<unique_hash>.lua

Install a package for different types of hardware nodes

Pawsey systems are made up of different types of hardware nodes; for instance, Setonix includes login, compute, data-mover and visualisation nodes.

If you need to install a package with Spack for usage on a specific node, the best way to proceed is to perform the installation on that specific type of node. For instance, to install applications for your HPC jobs, use a compute node via a Slurm allocation; to install a utility for the data-mover nodes, use a data-mover node.

Using Spack-installed software

Spack-installed software can be executed by any of these methods:

  • Calling the executable with the full path
  • Using spack load
  • Generating a modulefile with Spack and using it (recommended)

Using spack load

The load/unload command provides a simple fashion of updating the PATH environment variable to include software built by Spack. You can investigate the installed software using the find subcommand and then use the load command once you have identified the software to use. If several different versions of the package have been built by Spack, you will need to specify the desired spec:

Terminal 7: spack find and load example
$ spack find -vd fftw # list the variants and dependencies of fftw
-- linux-sles15-zen3 / gcc@11.2.0 -----------------------------
fftw@3.3.7~mpi~openmp~pfft_patches precision=double,float

fftw@3.3.7~mpi+openmp~pfft_patches precision=double,float

fftw@3.3.8~mpi~openmp~pfft_patches precision=double,float

$ spack load fftw # try loading any fftw
==> Error: fftw matches multiple packages.
  Matching packages:
    m4vsnhq fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    ax6d2kl fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    7mlxv4g fftw@3.3.8%gcc@11.2.0 arch=linux-sles15-zen3
  Use a more specific spec.
$ spack load fftw@3.3.7 # try loading a specific version
==> Error: fftw@3.3.7 matches multiple packages.
  Matching packages:
    m4vsnhq fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
    ax6d2kl fftw@3.3.7%gcc@11.2.0 arch=linux-sles15-zen3
  Use a more specific spec.
$ spack load fftw@3.3.7+openmp # load the version and variant that uniquely identifies the package

Once a path has been updated you can use the executable.  

Generating and using a modulefile

To generate modulefiles for Spack-built packages use the idiomatic syntax below. You will then be able to see them with module avail and use them with module load/unload.

The spack command shown in Terminal 8 will generate the modulefiles in the user location configured by Pawsey, which is:


In this example:

  • <cpu-arch> = zen3
  • <compiler-name> = gcc
  • <compiler-version> = 11.2.0

Terminal 8: spack module example
$ sg <projectcode> -c 'spack module lmod refresh --delete-tree -y'  # generate LUA modulefiles for Lmod
==> Regenerating lmod module files

$ # list the fftw modules available  
$ module avail fftw
$ # load a particular fftw library 
$ module load fftw/3.3.7-aormek7
$ # to determine if this module is the correct variant one can examine the module file itself 
$ head -4 /software/projects/pawsey0xxx/user/setonix/modules/zen3/gcc/11.2.0/fftw/3.3.7-aormek7.lua | tail -1 
-- fftw@3.3.7%gcc@11.2.0~mpi~openmp~pfft_patches precision=double,float arch=linux-sles15-zen3/aormek7
$ # compare this to other 3.3.7 module 
$ head -4 /software/projects/pawsey000x/user/setonix/modules/zen3/gcc/11.2.0/fftw/3.3.7-ibdoa45.lua | tail -1 
-- fftw@3.3.7%gcc@11.2.0~mpi+openmp+pfft_patches patches=8132c27659f992311dcf3d1500056e0f9400aa22f6824124e3607dbaa8dfe3c0 precision=double,float arch=linux-sles15-zen3/ibdoa45

Installing a software package from a user-modified source code

It is quite common for users to modify or patch the code of popular applications such as LAMMPS and Gromacs. You can use Spack to install the modified code as long as there exists a recipe for it (otherwise you can always create one).  Here is an example on how to install a modified Gromacs source code.

First, we need a Spack recipe that takes a tar file from the current working directory rather than the Internet:

$ cp -r /software/setonix/2022.11/spack/var/spack/repos/builtin/packages/gromacs $MYSOFTWARE/setonix/spack_repo/packages

Then open the file

$ vim $MYSOFTWARE/setonix/spack_repo/packages/gromacs/

And replace the url property with the following line

$ url = "file://{0}/gromacs.5.1.2.tar.gz".format(os.getcwd())

Now you can use spack to install your custom version. You will need to tar your code as if it were an official distribution, but name the tar file with a non-official version, for instance gromacs.5.1.2custom.tar.gz. Then run the usual spack install command:

$ spack install gromacs@5.1.2custom

in the same directory of the tar file.

Software built with Spack

Spack makes use of dynamic libraries and rpath when building software. The use of rpath  means that executables contain the paths in which libraries are search for at runtime. The result of using rpath  is that runtime are reproducible as executables will always point to the same library regardless of environment at runtime. A consequence of this is that if software has been built with not the desired version and build of library, it is not easy to have the executable use a different version. The package must be rebuilt with the different version of the library. 

The use of rpath  has a positive impact on the behaviour of executables provided by modules compared to Pre-Setonix systems (see Modules).

Learning more about Spack

Spack comes with a high-quality and detailed set of documentation and tutorial pages:

We recommend referring to these resources for information on topics that are not covered in this Pawsey Spack page, such as:

  • Spack configuration and customisation
  • Creation and debugging of packages
  • Use of Spack to develop an in-house code inside a Spack environment (developer workflow)

  • No labels