Page tree
Skip to end of metadata
Go to start of metadata

On this page:

Introduction - why using modules?


Our HPC systems have several tens of different installed packages, each one requiring its own set of environment variables to run properly. A module system is a simple and efficient way to keep the shell environment clean, by collecting variables related to each package in a separate unit, or module, that can be loaded only when required. Typical environment variables that are updated when loading a module include PATH, LD_LIBRARY_PATH and MANPATH; each package usually defines other specific variables, too.

Magnus/Galaxy and Zeus use two different packages for module management: Environment Modules and Lmod, respectively. Implementations have some differences, for instance modulefiles are typically written in TCL and LUA languages, respectively. Nonetheless, the user interface (commands and behaviours) are largely the same.

Common tasks handled by modules


There are three type of common operations executed by modules when loaded:

  1. Variable definition: see table below for a list of the most common variables and their use.
  2. Prerequisite module loading: these are package dependencies required by the application to work (scientific libraries, other packages).
  3. Declaration of conflicts: this is typically used to declare that a given package has not been compiled with a certain compiler or for a certain CPU architecture.
VariableScopeUse
PATHRuntime of the current packagePath to the executable
LD_LIBRARY_PATHRuntime of another package that links to the current library packagePath to the shared library
LIBRARY_PATHCompile time of another package that links to current library packagePath to the shared library
CPATHCompile time of another package that links to current library packagePath to the header files
MANPATHAnytimePath to documentation files loaded by "man"
Module specific pathsChanges

Some packages for instance need to know the path of their root installation folder

OMP_NUM_THREADSRuntime of the current packageDefault number of OpenMP threads

How to use modules


Modules are configured and ready to use upon login on all Pawsey HPC systems. Any supported software has its corresponding module.

To see a list of all the module commands use

module help

Here are the basic module commands:

  • module avail - list all the modules that are available on that machine (loaded and not loaded),
  • module list - list all of the loaded modules in your current shell environment,
  • module load module_name - load the specified module (e.g. application) to your environment,
  • module unload module_name - remove the specified module (e.g. application) from your environment,
  • module swap moduleA moduleB - unload moduleA and load moduleB,
  • module display module_name - output the content of the modulefile without loading it,
  • module whatis module_name - output summary information about the specified module (e.g. application) without loading it.

What modules are available: module avail

hostname-1:~> module avail
...

The module avail command will provide a full list of all the modules that are currently available in the current machine (but not necessarily loaded). Some module names appear multiple times with different version numbers, in which case one will be labelled as the default one.

What modules are currently loaded: module list

hostname-1:~> module list
...

The module list command provides a numbered list of the currently loaded modules, which are usually displayed in the form name/version. At login time, many of these are related to system software and are of no direct interest to the user.

Default modules on Magnus/Galaxy

magnus-1:~$ module list
Currently Loaded Modulefiles:
  1) slurm/17.02.9                                   5) craype-network-aries                            9) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari         13) xpmem/2.2.4-6.0.5.1_8.13__g35d5e73.ari         17) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari         21) cray-mpich/7.7.0
  2) pawseytools/1.20                                6) craype/2.5.13                                  10) pmi/5.0.13                                     14) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari           18) atp/2.1.1                                      22) craype-haswell
  3) modules/3.2.10.6                                7) cray-libsci/17.12.1                            11) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari         15) dvs/2.7_2.2.53-6.0.5.2_8.6__ge13250e           19) perftools-base/7.0.0
  4) cce/8.6.5                                       8) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari        12) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari  16) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari         20) PrgEnv-cray/6.0.4

Upon login, by default the Cray programming environment is loaded, PrgEnv-cray module, which in turn loads the module for the Cray compiler suite, cce, the MPI module, cray-mpich, and others. Note also that the SLURM scheduling system is loaded as a module.

Default modules on Zeus

zeus-1:~$ module list

Currently Loaded Modules:
  1) pawseytools/1.19   2) slurm/17.02.9   3) sandybridge/1.0   4) gcc/4.8.5

Default modules on Topaz

topaz-1:~$ module list

Currently Loaded Modules:
  1) cascadelake/1.0   2) pawseytools/1.28   3) slurm/20.02.3   4) gcc/8.3.0

Modules loaded at login that are relevant for the user include the GCC compiler module, gcc, and the SLURM module.

Please note that PawseyTools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff.

Loading, unloading, and swapping modules

To load a new module, use the module load command:

hostname-1:~> module load python

Note here python is specified without a version number, which means the default version is loaded. If a specific available version is wanted, we must unload the existing one and load the required version explicitly:

hostname-1:~> module unload python
hostname-1:~> module load python/3.6.3

A short-cut to replace the separate unload / load steps is available: module swap (or, equivalently, module switch). This is particularly useful in the context of switching between different programming environments:

magnus-1:~> module swap PrgEnv-cray PrgEnv-gnu

or compilers:

zeus-1:~> module swap gcc intel

Displaying information on modules

Further information of a descriptive nature can be obtained with the command module whatis:

hostname-1:~$ module whatis intel-mkl
intel-mkl            : Intel® Math Kernel Library (Intel® MKL) optimizes code with minimal effort
for future generations of Intel® processors. It is compatible with your choice
of compilers, languages, operating systems, and linking and threading models.

- Features highly optimized, threaded, and vectorized math functions that
  maximize performance on each processor family
- Utilizes industry-standard C and Fortran APIs for compatibility with popular
  BLAS, LAPACK, and FFTW functions-no code changes required
- Dispatches optimized code for each processor automatically without the need
  to branch code


For further information see https://software.intel.com/en-us/mkl

whereas the full content of the modulefile can be obtained using module display (or, equivalently, module show), providing information on the module dependencies and environment variables:

hostname-1:~$ module display intel-mkl
-------------------------------------------------------------------
/pawsey/cle60up05/modulefiles/devel/intel-mkl/2017.0.3:

module-whatis	 Intel® Math Kernel Library (Intel® MKL) optimizes code with minimal effort
for future generations of Intel® processors. It is compatible with your choice
of compilers, languages, operating systems, and linking and threading models.

- Features highly optimized, threaded, and vectorized math functions that
  maximize performance on each processor family
- Utilizes industry-standard C and Fortran APIs for compatibility with popular
  BLAS, LAPACK, and FFTW functions-no code changes required
- Dispatches optimized code for each processor automatically without the need
  to branch code


For further information see https://software.intel.com/en-us/mkl

 
setenv		 MAALI_SYSTEMINTELMKL_HOME /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl 
prepend-path	 INTEL_INC_PATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include 
prepend-path	 INTEL_LIB_PATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 
setenv		 INTEL_LICENSE_FILE /opt/intel/licenses/license.lic 
prepend-path	 LD_LIBRARY_PATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 
prepend-path	 MANPATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/man/en_US 
setenv		 MKLROOT /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl 
setenv		 MKL_DYNAMIC FALSE 
setenv		 MKL_HOME /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl 
setenv		 MKL_NUM_THREADS 1 
setenv		 OMP_NUM_THREADS 1 
prepend-path	 PATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/tools 
-------------------------------------------------------------------

A variant of module alias can be used to search modules whose name contains a specific string. On Magnus:

magnus-1:~$ module avail -S mpi

-------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/modulefiles --------------------------------------------------------------------------------------------------------------------------------------------------
cray-mpich/7.7.0(default)     cray-mpich-abi/7.7.0(default)

-------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/cle60up05/modulefiles/devel ---------------------------------------------------------------------------------------------------------------------------------------------
mpibash/1.3(default)

-------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/cle60up05/modulefiles/python --------------------------------------------------------------------------------------------------------------------------------------------
mpi4py/3.0.0(default)

On Zeus:

zeus-1:~$ module avail mpi

--------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/sles12sp3/modulefiles/apps ---------------------------------------------------------------------------------------------------------------------------------------------
   mpifileutils/master

-------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/sles12sp3/modulefiles/devel ---------------------------------------------------------------------------------------------------------------------------------------------
   intel-mpi/2017.0.3    intel-mpi/2017.0.4 (D)    mpibash/1.2    openmpi/2.1.2

-------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/sles12sp3/modulefiles/python --------------------------------------------------------------------------------------------------------------------------------------------
   mpi4py/3.0.0

  Where:
   D:  Default Module

Possible issue with module display on Zeus

zeus-1:~$ module display numpy
Lmod has detected the following error: Unable to load module:
     /pawsey/sles12sp3/modulefiles/python/numpy/1.13.3.lua: [string "help("Sets up the paths you need to use numpy..."]:24: attempt to concatenate a nil value

While processing the following module(s):
    Module fullname  Module Filename
    ---------------  ---------------
    numpy/1.13.3     /pawsey/sles12sp3/modulefiles/python/numpy/1.13.3.lua

This can happen with the Lmod environment on Zeus, typically when enquiring on modules that require Python or CUDA. The corresponding modules are not loaded at login, and a modulefile requiring them will contain a reference to some unset environment variables, causing an error.

In this case running following sequence of commands will solve the problem:

zeus-1:~$ module load python
zeus-1:~$ module display numpy

Moreover, the error message contains the full pathname of modulefile of interest, which can be displayed using cat:

zeus-1:~$ cat /pawsey/sles12sp3/modulefiles/python/numpy/1.13.3.lua

Usage examples


The most typical use of modules would be for loading applications to be run using the SLURM scheduler. In this case appropriate modules need to be loaded within the SLURM batch script:

#!/bin/bash -l
#SBATCH --account=pawsey0001
#SBATCH --partition=workq
#SBATCH --time=24:00:00
#SBATCH --nodes=2
#SBATCH --export=none
#SBATCH --job-name=lammps-job

module swap PrgEnv-cray PrgEnv-gnu
module load lammps

srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log

Here we want to run a simulation with the GCC version of the "lammps" package; we then use two module commands, to switch programming environment module and load the application module.

In other situations, we might need some lightweight tool on the interactive shell, e.g. python to run a post-processing analysis:

zeus-1:~$ module load python
zeus-1:~$ python average.py data.out

Or gnuplot for plotting:

zeus-1:~$ module load gnuplot
...
In principle, it would be possible to add module load commands in the shell initialisation scripts (~/.bashrc, ~/.tcshrc and so on). However, this practice makes the user environment less producible by colleagues and Pawsey staff, and is therefore strongly discouraged. See Best practices pages: Shell Initialisation Scripts and Jobscript Reproducibility.

System-wide, group and user modules


When operating with modules, the module environment looks for them along the sequence of directories listed in the MODULEPATH environment variable. For instance, this is the default MODULEPATH content on Zeus:

zeus-1:~$ echo $MODULEPATH 
/group/pawsey0001/mdelapierre/software/sles12sp3/modulefiles:/group/pawsey0001/software/sles12sp3/modulefiles:/opt/modulefiles:/opt/ohpc/pub/modulefiles:/usr/share/lmod/modulefiles:/pawsey/sles12sp3/modulefiles/apps:/pawsey/sles12sp3/modulefiles/bio-apps:/pawsey/sles12sp3/modulefiles/devel:/pawsey/sles12sp3/modulefiles/python:/pawsey/sles12sp3/modulefiles/tools

The sequence of directories in MODULEPATH is configured in such a way that the user specific module path is searched first, followed by the group specific path, and then by the system specific ones.

A user group can then install their own version of a package already available on the machine, and as a result their custom version will be selected instead of the system-wide one when running module load. Similarly, if a user installs his personal version of a package, this will have priority over both the group and the system-wide installations.

If a group or user wants to create their own module file for a custom installed application, they should locate and name it according to the following rule on Magnus and Galaxy:

/group/$MYGROUP/software/cle60up05/modulefiles/<software>/<version>			#group installation
/group/$MYGROUP/$USER/software/cle60up05/modulefiles/<software>/<version>	#user installation

And on Zeus:

/group/$MYGROUP/software/sles12sp3/modulefiles/<software>/<version>.lua			#group installation
/group/$MYGROUP/$USER/software/sles12sp3/modulefiles/<software>/<version>.lua	#user installation

If for whatever reason a custom directory needs to be used, then the dedicated module use command can be used to prepend that directory to the MODULEPATH variable:

<PATH-TO-MODULES>/<software>/<version>


magnus-1:~> module use <PATH-TO-MODULES>
magnus-1:~> module load <software>/<version>

Module Cache


The Lmod module environment used on Zeus utilises a module cache to speed up module commands.

To force the cache to update, remove the cache directory and run module avail:

rm -rf ~/.lmod.d/.cache
module avail



For more information, please refer to the following pages.

Pages in this section:



  • No labels