This tutorial illustrates how to set up a workflow that makes use of both CPU and GPU applications. It leverages a custom Nextflow build to provide support for Slurm multiple clusters. The feature is currently not supported by the Nextflow development team.

Context

Suppose you have a Nextflow pipeline where some processes make use of CPU applications, while other make use of GPU applications. Suppose also that you are using containers to deploy these applications by means of Singularity. There are a few aspects that need to be addressed to run this type of pipeline at Pawsey:

• CPU applications need to run on Setonix Phase 1, whereas GPU applications need to run on Topaz
• Nextflow needs to ask GPU resources to Slurm, in the case of GPU enabled processes
• Nextflow needs to enable GPU usage for Singularity

Details

• To submit tasks to both Setonix and Topaz, Nextflow needs a custom build to enable multi-cluster support. Detailed steps and a template script to build a customised Nextflow in the current context are provided in the Appendix on this page. In this example of customised Nextflow, two new executors are made available, slurm_setonix and slurm_topaz, which allow to assign jobs to Setonix and Topaz, respectively.
• Nextflow processes that make use of a CPU require these configuration parameters:
  • executor = 'slurm_setonix'  (valid for this example of customised Nextflow)
  • queue = 'work'
• Nextflow processes that make use of a GPU require these configuration parameters:
  • executor = 'slurm_topaz'  (valid for this example of customised Nextflow)
  • queue = 'gpuq'
  • clusterOptions += " --gpus-per-node=1"
• Special care is needed in the setup of Singularity environment variables SINGULARITY_BINDPATH, SINGULARITYENV_LD_LIBRARY_PATH and SINGULARITYENV_LD_PRELOAD, inherited from the Singularity module, because the master pipeline running on Setonix will execute tasks on distinct clusters with distinct installation paths. As a result, it is best NOT to pass the module variables at all, that is, not to include them in the envWhitelist parameter in the Nextflow configuration.
• The Singularity configuration in Nextflow requires the following:
  • runOptions = "-B /scratch --nv"
  • Here, --nv enables use of GPUs (this is compatible with CPU tasks, too, the only side effect is that a warning will be printed)
  • Here, -B /scratch is also added, because SINGULARITY_BINDPATH is not whitelisted from the Singularity module (see point above)

Example

  setonixtopaz {
  
     process.container = 'marcodelapierre/toy-gpu-nf:latest'
     singularity {
       enabled = true
       runOptions = "-B /scratch --nv"
     }
    
    params.slurm_account = 'pawsey0001'
    process {
      clusterOptions = "--account=${params.slurm_account}"
      executor = 'slurm_setonix'
      queue = 'work'

      withName: 'proc_gpu' {
        executor = 'slurm_topaz'
        queue = 'gpuq'
        clusterOptions += " --gpus-per-node=1"
      }
    }
  
  }

Appendix

Details on building Nexflow with multi-cluster support

The idea is to duplicate the existing slurm executor, to create additional ones that submit to specific Pawsey supercomputers, Setonix and Topaz in this case. This new executor will have modified sbatch, squeue and scancel commands, using e.g. the --clusters topaz flag. In this way, two Slurm executables are available to the Nextflow user, for submitting processes either to the current cluster, or to Topaz. The source directory, modules/nextflow/src/main/groovy/nextflow/executor/, contains relevant source files, in particular:

• ExecutorFactory.groovy - has the list of available executors. It's possible to add new ones
• SlurmExecutor.groovy - codes the standard slurm executor. E.g. it can be duplicated into a modified SlurmTopazExecutor.groovy

Here are the detailed steps:

1. Clone the Nextflow Github repository, and check out the desired version
2. Edit the file modules/nextflow/src/main/groovy/nextflow/executor/ExecutorFactory.groovy. Locate the class ExecutorFactory, then add additional Slurm executors in the list of executors, by naming a new class file, e.g. SlurmTopazExecutor
3. In the same directory, duplicate the class file SlurmExecutor.groovy into a new file, e.g. SlurmTopazExecutor.groovy. In the new file, locate the occurrences of sbatch, squeue and scancel (one each), and add the clusters flag with appropriate Groovy syntax. In this example, '--clusters', 'topaz' 
4. Compile your customised Nextflow with the sequence
   $ make compile; "make pack; make install
5. Retrieve the compiled executable from nextflow/build/releases/nextflow-*-all

Template to build Nexflow with multi-cluster support

#!/bin/bash
 
# assuming this script is run on Setonix
 
 
# CUSTOMISE : nextflow version we want to start with
nxf_version="v22.04.3"
 

# this is good habit on Pawsey systems
export NXF_HOME="$MYSOFTWARE/.nextflow"
# this is needed only when building Nextflow
export GRADLE_USER_HOME="$MYSOFTWARE/.gradle"

# working directories
here=$(pwd)
tmp_dir="/tmp/tmp_nextflow_build_$((RANDOM))"

echo ""
echo "building nextflow in directory $tmp_dir"
echo"" 
mkdir -p $tmp_dir
cd $tmp_dir
 
# get nextflow repo to the right version
git clone https://github.com/nextflow-io/nextflow
cd nextflow
git checkout $nxf_version


# CUSTOMISE: add Slurm executors as needed
# edit the code
# assuming the new executors are called "slurm_setonix" and "slurm_topaz"
sed -i 's/{HOME}/{MYSOFTWARE}/g' Makefile
 
sed -i "/SlurmExecutor/ a\            'slurm_setonix': SlurmSetonixExecutor," \
    modules/nextflow/src/main/groovy/nextflow/executor/ExecutorFactory.groovy
 
sed -i "/SlurmExecutor/ a\            'slurm_topaz': SlurmTopazExecutor," \
    modules/nextflow/src/main/groovy/nextflow/executor/ExecutorFactory.groovy
 
sed -e 's/class SlurmExecutor/class SlurmSetonixExecutor/g' \
    -e "s/'sbatch'/&, '--clusters', 'setonix'/g" \
    -e "s/'squeue'/&, '--clusters', 'setonix'/g" \
    -e "s/'scancel'/&, '--clusters', 'setonix'/g" \
    modules/nextflow/src/main/groovy/nextflow/executor/SlurmExecutor.groovy >modules/nextflow/src/main/groovy/nextflow/executor/SlurmSetonixExecutor.groovy
 
sed -e 's/class SlurmExecutor/class SlurmTopazExecutor/g' \
    -e "s/'sbatch'/&, '--clusters', 'topaz'/g" \
    -e "s/'squeue'/&, '--clusters', 'topaz'/g" \
    -e "s/'scancel'/&, '--clusters', 'topaz'/g" \
    modules/nextflow/src/main/groovy/nextflow/executor/SlurmExecutor.groovy >modules/nextflow/src/main/groovy/nextflow/executor/SlurmTopazExecutor.groovy


# compile
module load openjdk/17.0.0_35
 
make compile
make pack
make install
 
# retrieve nextflow executable
cd $here
cp -p $tmp_dir/nextflow/build/releases/nextflow-*-all nextflow
chmod +x nextflow
 
# test nextflow executable
./nextflow info
ok="$?"
 
# clean build dir
rm -rf $tmp_dir

if [ "$ok" == "0" ] ; then 
# print final information
  echo ""
  echo "edited nextflow made available at $here/nextflow"
  echo ""
  echo "to use it, add these variables to your shell environment, or to your .bashrc or .bash_profile"
  echo "module load openjdk/17.0.0_35"
  echo "export NXF_HOME=\"\$MYSOFTWARE/.nextflow\""
  echo "export PATH=\"$here:\$PATH\""
  echo ""
else
  echo ""
  echo "something went wrong in the installation. OOPS."
fi

exit

Related page

• Nextflow