Introduction - why using modules?
Our HPC systems have several tens of different installed packages, each one requiring its own set of environment variables to run properly. A module system is a simple and efficient way to keep the shell environment clean, by collecting variables related to each package in a separate unit, or module, that can be loaded only when required. Typical environment variables that are updated when loading a module include PATH, LD_LIBRARY_PATH and MANPATH; each package usually defines other specific variables, too.
Magnus/Galaxy and Zeus use two different packages for module management: Environment Modules and Lmod, respectively. Implementations have some differences, for instance modulefiles are typically written in TCL and LUA languages, respectively. Nonetheless, the user interface (commands and behaviours) are largely the same.
Common tasks handled by modules
There are three type of common operations executed by modules when loaded:
- Variable definition: see table below for a list of the most common variables and their use.
- Prerequisite module loading: these are package dependencies required by the application to work (scientific libraries, other packages).
- Declaration of conflicts: this is typically used to declare that a given package has not been compiled with a certain compiler or for a certain CPU architecture.
| Variable | Scope | Use |
|---|---|---|
| PATH | Runtime of the current package | Path to the executable |
| LD_LIBRARY_PATH | Runtime of another package that links to the current library package | Path to the shared library |
| LIBRARY_PATH | Compile time of another package that links to current library package | Path to the shared library |
| CPATH | Compile time of another package that links to current library package | Path to the header files |
| MANPATH | Anytime | Path to documentation files loaded by "man" |
| Module specific paths | Changes | Some packages for instance need to know the path of their root installation folder |
| OMP_NUM_THREADS | Runtime of the current package | Default number of OpenMP threads |
How to use modules
Modules are configured and ready to use upon login on all Pawsey HPC systems. Any supported software has its corresponding module.
To see a list of all the module commands use
module help
Here are the basic module commands:
- module avail - list all the modules that are available on that machine (loaded and not loaded),
- module list - list all of the loaded modules in your current shell environment,
- module load module_name - load the specified module (e.g. application) to your environment,
- module unload module_name - remove the specified module (e.g. application) from your environment,
- module swap moduleA moduleB - unload moduleA and load moduleB,
- module display module_name - output the content of the modulefile without loading it,
- module whatis module_name - output summary information about the specified module (e.g. application) without loading it.
What modules are available: module avail
hostname-1:~> module avail ...
The module avail command will provide a full list of all the modules that are currently available in the current machine (but not necessarily loaded). Some module names appear multiple times with different version numbers, in which case one will be labelled as the default one.
What modules are currently loaded: module list
hostname-1:~> module list ...
The module list command provides a numbered list of the currently loaded modules, which are usually displayed in the form name/version. At login time, many of these are related to system software and are of no direct interest to the user.
Default modules on Magnus/Galaxy
magnus-1:~$ module list Currently Loaded Modulefiles: 1) slurm/17.02.9 5) craype-network-aries 9) ugni/6.0.14-6.0.5.0_16.9__g19583bb.ari 13) xpmem/2.2.4-6.0.5.1_8.13__g35d5e73.ari 17) rca/2.2.16-6.0.5.0_15.34__g5e09e6d.ari 21) cray-mpich/7.7.0 2) pawseytools/1.20 6) craype/2.5.13 10) pmi/5.0.13 14) job/2.2.2-6.0.5.0_8.47__g3c644b5.ari 18) atp/2.1.1 22) craype-haswell 3) modules/3.2.10.6 7) cray-libsci/17.12.1 11) dmapp/7.1.1-6.0.5.0_49.8__g1125556.ari 15) dvs/2.7_2.2.53-6.0.5.2_8.6__ge13250e 19) perftools-base/7.0.0 4) cce/8.6.5 8) udreg/2.3.2-6.0.5.0_13.12__ga14955a.ari 12) gni-headers/5.0.12-6.0.5.0_2.15__g2ef1ebc.ari 16) alps/6.5.28-6.0.5.0_18.6__g13a91b6.ari 20) PrgEnv-cray/6.0.4
Upon login, by default the Cray programming environment is loaded, PrgEnv-cray module, which in turn loads the module for the Cray compiler suite, cce, the MPI module, cray-mpich, and others. Note also that the SLURM scheduling system is loaded as a module.
Default modules on Zeus
zeus-1:~$ module list Currently Loaded Modules: 1) pawseytools/1.19 2) slurm/17.02.9 3) sandybridge/1.0 4) gcc/4.8.5
Default modules on Topaz
topaz-1:~$ module list Currently Loaded Modules: 1) cascadelake/1.0 2) pawseytools/1.28 3) slurm/20.02.3 4) gcc/8.3.0
Modules loaded at login that are relevant for the user include the GCC compiler module, gcc, and the SLURM module.
Please note that PawseyTools module is also loaded by default on all Pawsey systems. This module contains a collection of useful utilities provided by Pawsey staff.
Loading, unloading, and swapping modules
To load a new module, use the module load command:
hostname-1:~> module load python
Note here python is specified without a version number, which means the default version is loaded. If a specific available version is wanted, we must unload the existing one and load the required version explicitly:
hostname-1:~> module unload python hostname-1:~> module load python/3.6.3
A short-cut to replace the separate unload / load steps is available: module swap (or, equivalently, module switch). This is particularly useful in the context of switching between different programming environments:
magnus-1:~> module swap PrgEnv-cray PrgEnv-gnu
or compilers:
zeus-1:~> module swap gcc intel
Displaying information on modules
Further information of a descriptive nature can be obtained with the command module whatis:
hostname-1:~$ module whatis intel-mkl intel-mkl : Intel® Math Kernel Library (Intel® MKL) optimizes code with minimal effort for future generations of Intel® processors. It is compatible with your choice of compilers, languages, operating systems, and linking and threading models. - Features highly optimized, threaded, and vectorized math functions that maximize performance on each processor family - Utilizes industry-standard C and Fortran APIs for compatibility with popular BLAS, LAPACK, and FFTW functions-no code changes required - Dispatches optimized code for each processor automatically without the need to branch code For further information see https://software.intel.com/en-us/mkl
whereas the full content of the modulefile can be obtained using module display (or, equivalently, module show), providing information on the module dependencies and environment variables:
hostname-1:~$ module display intel-mkl ------------------------------------------------------------------- /pawsey/cle60up05/modulefiles/devel/intel-mkl/2017.0.3: module-whatis Intel® Math Kernel Library (Intel® MKL) optimizes code with minimal effort for future generations of Intel® processors. It is compatible with your choice of compilers, languages, operating systems, and linking and threading models. - Features highly optimized, threaded, and vectorized math functions that maximize performance on each processor family - Utilizes industry-standard C and Fortran APIs for compatibility with popular BLAS, LAPACK, and FFTW functions-no code changes required - Dispatches optimized code for each processor automatically without the need to branch code For further information see https://software.intel.com/en-us/mkl setenv MAALI_SYSTEMINTELMKL_HOME /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl prepend-path INTEL_INC_PATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/include prepend-path INTEL_LIB_PATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 setenv INTEL_LICENSE_FILE /opt/intel/licenses/license.lic prepend-path LD_LIBRARY_PATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/lib/intel64 prepend-path MANPATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/man/en_US setenv MKLROOT /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl setenv MKL_DYNAMIC FALSE setenv MKL_HOME /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl setenv MKL_NUM_THREADS 1 setenv OMP_NUM_THREADS 1 prepend-path PATH /opt/intel/compilers_and_libraries_2017.4.196/linux/mkl/tools -------------------------------------------------------------------
A variant of module alias can be used to search modules whose name contains a specific string. On Magnus:
magnus-1:~$ module avail -S mpi -------------------------------------------------------------------------------------------------------------------------------------------------- /opt/cray/pe/modulefiles -------------------------------------------------------------------------------------------------------------------------------------------------- cray-mpich/7.7.0(default) cray-mpich-abi/7.7.0(default) -------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/cle60up05/modulefiles/devel --------------------------------------------------------------------------------------------------------------------------------------------- mpibash/1.3(default) -------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/cle60up05/modulefiles/python -------------------------------------------------------------------------------------------------------------------------------------------- mpi4py/3.0.0(default)
On Zeus:
zeus-1:~$ module avail mpi --------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/sles12sp3/modulefiles/apps --------------------------------------------------------------------------------------------------------------------------------------------- mpifileutils/master -------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/sles12sp3/modulefiles/devel --------------------------------------------------------------------------------------------------------------------------------------------- intel-mpi/2017.0.3 intel-mpi/2017.0.4 (D) mpibash/1.2 openmpi/2.1.2 -------------------------------------------------------------------------------------------------------------------------------------------- /pawsey/sles12sp3/modulefiles/python -------------------------------------------------------------------------------------------------------------------------------------------- mpi4py/3.0.0 Where: D: Default Module
Possible issue with module display on Zeus
zeus-1:~$ module display numpy
Lmod has detected the following error: Unable to load module:
/pawsey/sles12sp3/modulefiles/python/numpy/1.13.3.lua: [string "help("Sets up the paths you need to use numpy..."]:24: attempt to concatenate a nil value
While processing the following module(s):
Module fullname Module Filename
--------------- ---------------
numpy/1.13.3 /pawsey/sles12sp3/modulefiles/python/numpy/1.13.3.lua
This can happen with the Lmod environment on Zeus, typically when enquiring on modules that require Python or CUDA. The corresponding modules are not loaded at login, and a modulefile requiring them will contain a reference to some unset environment variables, causing an error.
In this case running following sequence of commands will solve the problem:
zeus-1:~$ module load python zeus-1:~$ module display numpy
Moreover, the error message contains the full pathname of modulefile of interest, which can be displayed using cat:
zeus-1:~$ cat /pawsey/sles12sp3/modulefiles/python/numpy/1.13.3.lua
Usage examples
The most typical use of modules would be for loading applications to be run using the SLURM scheduler. In this case appropriate modules need to be loaded within the SLURM batch script:
#!/bin/bash -l #SBATCH --account=pawsey0001 #SBATCH --partition=workq #SBATCH --time=24:00:00 #SBATCH --nodes=2 #SBATCH --export=none #SBATCH --job-name=lammps-job module swap PrgEnv-cray PrgEnv-gnu module load lammps srun --export=all -n 48 lmp_mpi -in lammps.inp -log lammps.log
Here we want to run a simulation with the GCC version of the "lammps" package; we then use two module commands, to switch programming environment module and load the application module.
In other situations, we might need some lightweight tool on the interactive shell, e.g. python to run a post-processing analysis:
zeus-1:~$ module load python zeus-1:~$ python average.py data.out
Or gnuplot for plotting:
zeus-1:~$ module load gnuplot ...
System-wide, group and user modules
When operating with modules, the module environment looks for them along the sequence of directories listed in the MODULEPATH environment variable. For instance, this is the default MODULEPATH content on Zeus:
zeus-1:~$ echo $MODULEPATH /group/pawsey0001/mdelapierre/software/sles12sp3/modulefiles:/group/pawsey0001/software/sles12sp3/modulefiles:/opt/modulefiles:/opt/ohpc/pub/modulefiles:/usr/share/lmod/modulefiles:/pawsey/sles12sp3/modulefiles/apps:/pawsey/sles12sp3/modulefiles/bio-apps:/pawsey/sles12sp3/modulefiles/devel:/pawsey/sles12sp3/modulefiles/python:/pawsey/sles12sp3/modulefiles/tools
The sequence of directories in MODULEPATH is configured in such a way that the user specific module path is searched first, followed by the group specific path, and then by the system specific ones.
A user group can then install their own version of a package already available on the machine, and as a result their custom version will be selected instead of the system-wide one when running module load. Similarly, if a user installs his personal version of a package, this will have priority over both the group and the system-wide installations.
If a group or user wants to create their own module file for a custom installed application, they should locate and name it according to the following rule on Magnus and Galaxy:
/group/$MYGROUP/software/cle60up05/modulefiles/<software>/<version> #group installation /group/$MYGROUP/$USER/software/cle60up05/modulefiles/<software>/<version> #user installation
And on Zeus:
/group/$MYGROUP/software/sles12sp3/modulefiles/<software>/<version>.lua #group installation /group/$MYGROUP/$USER/software/sles12sp3/modulefiles/<software>/<version>.lua #user installation
If for whatever reason a custom directory needs to be used, then the dedicated module use command can be used to prepend that directory to the MODULEPATH variable:
<PATH-TO-MODULES>/<software>/<version> magnus-1:~> module use <PATH-TO-MODULES> magnus-1:~> module load <software>/<version>
Module Cache
The Lmod module environment used on Zeus utilises a module cache to speed up module commands.
To force the cache to update, remove the cache directory and run module avail:
rm -rf ~/.lmod.d/.cache module avail
For more information, please refer to the following pages.