Page tree
Skip to end of metadata
Go to start of metadata


On this page:

Overview


OpenFOAM is a popular open-source Computational Fluid Dynamics (CFD) toolkit, written in C++, and based on the finite volume method.  Currently OpenFOAM-5.0 and OpenFOAM+ v1712 are installed on Pawsey systems.

  • OpenFOAM is developed by the OpenFOAM Foundation (https://openfoam.org), and aims to provide a stable, high-quality CFD toolkit.
  • OpenFOAM+ is developed by ESI (https://openfoam.com), and releases it's own fork of OpenFOAM on a different schedule.

Both versions are stable, portable, and widely used, but each does offer different options.  For example, OpenFOAM 5.0 has recently incorporated wave modelling solvers and utilities into its codebase, while OpenFOAM+ has not (although it's likely they will be added in later releases). OpenFOAM+ has more "bleeding-edge" options, such as support for newer decomposition methods like KaHIP or the parallel I/O format ADIOS. 

Running OpenFOAM


On Magnus

Example jobscript:

#!/bin/bash --login

#SBATCH --job-name="name_of_job"
#SBATCH --nodes=2
#SBATCH --time=1:00:00
#SBATCH --export=NONE

module swap PrgEnv-cray PrgEnv-intel
module load openfoam+

blockMesh
setFields
decomposePar
srun --export=all -N 2 -n 48 interFoam -parallel

This will use the Intel build of OpenFOAM+, and will run on 2 nodes with a total of 48 MPI tasks. Note that srun is only being used for the parallel OpenFOAM applications; your job is running on the compute node so srun isn't required for the serial tasks, only for ensuring proper placement of parallel applications.

On Zeus

Example jobscript for Broadwell nodes:

#!/bin/bash --login

#SBATCH --job-name="name_of_job"
#SBATCH --nodes=1
#SBATCH --ntasks=28
#SBATCH --time=1:00:00
#SBATCH --export=NONE

module swap sandybridge broadwell
module swap gcc intel
module load openfoam+

blockMesh
setFields
decomposePar
srun --export=all -N 1 -n 28 interFoam -parallel


Alternatively, because Zeus now allows shared nodes, you can also use Zeus for serial tasks such as data reconstruction: 

#!/bin/bash --login

#SBATCH --job-name="name_of_job"
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --time=1:00:00
#SBATCH --export=NONE

module swap sandybridge broadwell
module swap gcc intel
module load openfoam+

reconstructPar

Parallel I/O in OpenFOAM


-fileHandler collated

Since OpenFOAM-5.0 and OpenFOAM-v1706, a method for writing data in parallel (called collated I/O) was introduced.  Traditionally, parallel simulations in OpenFOAM generate a processor directory for each MPI task, and a number of time steps directories within each processor directory.  The basic collated method, instead, generates a single processors directory, with separate time step directories located in there.  Data for each decomposed field is collected by a master MPI task and written to a single file.  The result is far fewer files; 1 per field as opposed to N per field (where N is the number of processors). We have set this option as the default for installed versions.

Users are strongly discouraged from changing the default file handler, as uncollated OpenFOAM simulations can produce an extremely large number of files.  Pawsey staff may lock user accounts and prevent jobs from being run if they find users have a large number of files on the scratch filesystem.  Accounts will only be reinstated if files are removed.

There are 3 file handlers available in OpenFOAM

  • collated (recommended)
    • Fields are written to a single file by a master process
  • uncollated
    • The standard way of writing files, with each processor creating its own directory
  • masterUncollated
    • A version of uncollated.  Multiple processor directories are still created, but all I/O is directed through a master processor.  This handler is quite slow.

There are several ways for selecting the file handler (in increasing order of priority):

  1. Raw openfoam default = uncollated (not applicable for Pawsey due to option No. 3)
  2. Environmental variable "FOAM_FILEHANDLER" (not applicable to Pawsey due to option No. 3)
  3. etc/controlDict optimisationSwitches 'fileHandler' (option used at Pawsey to set the default to "collated")
  4. Add the following to their own $FOAM_CASE/system/controlDict:

    OptimisationSwitches
    {
    	fileHandler collated;
    }
  5. Using the command line option -fileHandler:

    srun --export=all -n 1 decomposePar -fileHandler collated

    For OpenFOAM-6 and -7, option 4. has a higher priority than option 5. This does not correspond to what is indicated here and in the official OpenFOAM documentation. We believe this is an error, and the priority order documented here is the way it should be. The mismatch has been reported to OpenFOAM Foundation developers already.

The collated fileHandler will produce a directory structure of the format processorsN, where N is the total number of processors.

Notes regarding the collated file format:

  • Binary output (without compression) should be used (although ASCII is supported)
  • ASCII collated files are not editable
  • Collated writing works best using a threaded MPI

Users can also convert between collated and uncollated formats with the foamFormatConvert utility:

srun --export=all -n 4 foamFormatConvert -parallel -fileHandler collated

-ioRanks '(0 G 2G 3G ... mG)'

When only the basic collated option (previous subsection) is used, performance issues may appear for a large number of parallel "processors". Problems arise because all IO communications are routed through a single processor. Since OpenFOAM-6 and OpenFOAM-v1806, the -fileHandler collated option was dramatically improved with the incorporation of the additional -ioRanks option. With the additional option, users can specify a list of "processors" (identified by their rank number) to be used for IO, and not just a single processor collating everything. The first processor in the list will group the IO of all the following processors of the simulation in increasing order until reaching the rank of the second processor in the list. The second processor in the list will collate the output of the following group of processor of the simulation in increasing order until reaching the rank of the third processor in the list, and so on. Commonly, the ioRanks list use an increment G (as indicated in the title of this section) grouping IO of every G simulation processors. Note that numbering of processors for OpenFOAM start with "0" (following c++ standards).

This additional feature needs to be used with the standard basic collated option (explained in the previous subsection). Once the "-fileHandler collated" option is set, users have only two ways for defining the ioRanks. Unfortunately, this additional option cannot be defined in the controlDict dictionary. The only two ways of definition, in increasing order of priority, are:

  1. Define the environmental variable FOAM_IORANKS (exemplified here for a pimpleFoam parallel run with 96 processors and collated every group of 24):

    export FOAM_IORANKS='(0 24 48 72 96 120 144 168 192 216 240 264 288 312 336 360 384 408 432 456 480 504 528)'
    srun --export=all -n 1 decomposePar
    srun --export=all -n 96 pimpleFoam -parallel

    The environment variable FOAM_IORANKS will be picked up by the decomposePar and the solver (pimpleFoam). In this example it is considered that the first part of the combined setting (collated option) has been set by default or inside the system/controlDict dictionary (see explanation in the previous subsection). Those numbers in the ioRanks list that are greater than the real number of processors to be used in the simulation (numberOfSubdomains set in the system/decomposeParDict, 96 in this case) will be ignored. So users can set a large ioRanks list first and then use any number of processsors in their simulations without any issues. Except when the numberOfSubdomains is less than the increment G in the ioRanks list; in that case, the ioRanks should not be set.

  2. Use the command line option (exemplified here for a pimpleFoam parallel run with 96 processors and collated every group of 24) :

    srun --export=all -n 1 decomposePar -fileHandler collated -ioRanks '(0 24 48 72)'
    srun --export=all -n 96 pimpleFoam -parallel -fileHandler collated -ioRanks '(0 24 48 72)'

    In this second example, it is considered that the first part of the combined setting (collated option) is not already set via dictionary or by default. Then, the whole combined setting is explictly indicated in the command line. If the first part were set previously (as in the example in point 1. above), then the "-fileHandler collated" part could have been ommited.

Processor "0" is the "master" processor and needs always to be the first in the list.

For the ioRanks option to work properly, the total number of processors used in the simulation (numberOfSubdomains) should be greater than the increment G used in the ioRanks list.

The use of ioRanks will produce a directory structure of the format processorsN_X-Y, where N is the total number of processors in the simulation, and X and Y are the first and last processor in each IO group. The decomposition examples in "points 1. & 2." above will produce the following directories:

processors96_0-23
processors96_24-47
processors96_48-71
processors96_72-95

Use 1 collating processor PER NODE

For the use of parallel solvers in Magnus: we recommend to list ioRanks in groups of 24 processors (i.e. one collating processor per node).
For the use of parallel sovers in Zeus: we recommend to list ioRanks in groups of 28 processors (i.e. one collating processor per node).


For more information refer to

https://www.openfoam.com/releases/openfoam-v1712/parallel.php

https://openfoam.org/news/parallel-io/

There are also fileHandler tutorials available in the OpenFOAM tutuorials:

$FOAM_TUTORIALS/IO/fileHandler

Post-processing in OpenFOAM


OpenFOAM has changed the interface for its post-processing tools, beginning with version 4.0. All non-gui utilities (e.g., paraFoam) have been combined into a single command line interface.  This allows user to control how and when data processing occurs.  Post-processing in OpenFOAM can broken into 2 methods:

  • A conventional approach where data is processed after a simulation has run.  This occurs via the unified command line interface, postProcess, provided by OpenFOAM.
  • Run-time processing, where a data is processed during the course of a simulation.  This requires a user to set up the appropriate functions in their simulation's controlDict.

Run-time processing is convenient as it allows users to monitor processed data as a simulation progresses, and it also provides users with processed results as soon as the simulation finishes.  However, this can increase the overall simulation and may cause jobs to abort when they reach the 24 hour time limit.

Users can post-process data in 3 different ways:


  • Configuring their solvers to use run-time processing
  • Using the postProcess utility in a separate job, operating on data written by a completed simulation
  • Passing the -postProcess option to a solver.  The solver only executes the post-processing function, but has access to other information related to that particular solver (e.g. the turbulence model used)

Pawsey recommends that users use the conventional post-processing method, and run post-processing commands on Zeus via the postProcess utility, after a computational run has completed on Magnus.  Zeus has shared nodes, so users can choose to run post-processing utilities on a desired number of cores, whereas running them on Magnus will result in users being charged for all available cores on a node, whether or not they are all used. 

Post-processing utilities

Once a simulation is complete, users can use the postProcess utility for post-processing needs.  All of the post-processing utilities can accessed via a function object:

postProcess -func "mag(U)"

This operation will compute the magnitude of the velocity field U, and write it to a filed name mag(U).  

postProcess -func "totalPressureIncompressible(p,U)"

Some utilities will require multiple fields to be loaded in order to work.  This command will compute the total pressure field for an incompressible solver.

Users can query the available post-processing utilities with the command:

postProcess -list

Run-time processing

To process data during a simulation a user needs to configure the simulations controlDict by adding the desired post-processing function to the functions section of the controlDict.  For example, if a user wanted to compute the magnitude of a field, controlDict would require the following:

functions
{
    #includeFunc mag(U)
} 

All of the configuration files for the various post-processing functions can be found in $FOAM_ETC/caseDicts/postProcessing.

Solver processing

OpenFOAM's solvers can also function as post-processing utilities if the the -postProcess flag is passed.  This results in only the post-processing utilities being executed.  This may be necessary if parts of the model are required to post-process.  For example, calculating wall shear stress with just the post-processing utility will produce the following:

postProcess -fields "(p U)" -func wallShearStress

–> FOAM FATAL ERROR: 
    Unable to find turbulence model in the database 

The turbulence model used in the simulation is required to compute wall shear stress, so the solver must be used in order to load the turbulence modelling data and make use of it during the post-processing:

simpleFoam -postProcess -func wallShearStress

Users can get help with using solver as a post-processing utility by passing the -help flag:

simpleFoam -postProcess -help

More information can be found at 

https://cfd.direct/openfoam/user-guide/post-processing-cli/

A number of the standard OpenFOAM tutorials also incorporate the post-processing methods (they can be found at $FOAM_TUTORIALS).  Several basic examples include:

  • basic/potentialFoam/pitzDaily
  • basic/potentialFoam/cylinder
  • incompressible/simpleFoam/pitzDaily

Best practices for using OpenFOAM


Introduction

The open-source CFD software package OpenFOAM is capable of utilizing MPI and is well-suited for use on a parallel system like Magnus. However, there are some features of OpenFOAM that many users overlook that could help improve the performance of their OpenFOAM applications.

OpenFOAM and the Filesystem

One of the issues with OpenFOAM is that it generates a large number of small files, and parallel filesystems work optimally with large files. In a multi-processor OpenFOAM simulation, each processor generates a new directory with multiple output files, and this process is repeated for each time step. For smaller simulations this is generally not an issue, but larger simulations have the potential to produce thousands and thousands of files.

For example, a 5 node job (120 processors) producing 875 time steps will result in 630,000 individual files. Each of the files results in I/O operations such as a read, write, open, close, or stat (returning information about a file). A large number of I/O operations will have an impact on both the user's application, as well as other applications running on the system. Several of the default options in OpenFOAM cases will result in an increase in I/O operations, and some tips are presented here to help reduce I/O load and improve performance.

Tips for running OpenFOAM

  • Use ~100,000 cells per processor
    As a rule of thumb, we recommend to assign around 100,000 computational cells per core for parallel simulations. This number is not written in stone, but it has proven to give good performance on Magnus. From this starting point, users can then test and modify their decomposition looking for better performance.
  • Time-stepping
    OpenFOAM writes output at each time-step, so a smaller time-step will result in a large number of files being written and I/O operations. Ideally, a user would set their time-step to largest possible for the simulation. However, some of the OpenFOAM solvers can make use of adaptive time-stepping (using larger time-steps when possible and scaling it as need). Several of the parameters in the controlDict dictionary that control adaptive time-stepping are:
    • adjustTimeStep yes|no
      Adjusts the time step, usually based on the Courant number
    • maxCo val
      Maximum Courant number to use (e.g., maxCo 0.5)
    • maxDeltaT val
      Maximum time step to use (e.g., maxDeltaT 0.1)
  • purgeWrite
    This is a controlDict parameter that controls how time directories are stored and overwritten. For example, with purgeWrite 2, data from time steps will be written into 2 directories, i and i+1. Data from subsequent time steps will overwrite these directories in a cyclical pattern; that is, time step i+2 will write to directory i, time step i+3 will write to directory i+1, etc. For steady-state problems, only 1 directory is necessary and previous time step data can be overwritten, so a value of purgeWrite 1 should be used. To turn off the limit, use purgeWrite 0.
  • File modifications
    OpenFOAM allows users to modify parameters while simulations are running. While this is useful for testing and debugging purposes, it requires OpenFOAM to monitor each file during execution. With multiple processors generating multiple files, this can result in thousands of stat I/O operations at each time step. It is hightly recommended that users disable file modifications. This can be accomplished with the following controlDict parameter:
  • runTimeModifiable yes|no
    OpenFOAM simulations tend to have a large overhead in the startup phase; the runTimeModifiable flag is a major factor in this behavior. Turning off this option can greatly reduce this initial overhead and result in faster simulations.
  • Binary vs. ASCII output
    OpenFOAM defaults to writing output in ASCII format. While this allows users to easily read and edit files, it does affect performance (particularly with large simulations). Writing output in binary format is faster than writing in ASCII and users are encouraged to use binary. To control the format of output in OpenFOAM, users can use the following controlDict parameter: writeFormat ascii|binary
  • -fileHandler collated -ioRanks '(0 G 2G ... mG)'
    Use the collated fileHandler to reduce the number of result directories and files. Collate the output in groups of G processors, where G is the number of processors (cores) available per node.
  • File curation
    Users are strongly encouraged to delete files on scratch when they are no longer needed, rather than simply wait for file purging to remove them.  Because of OpenFOAM's popularity on our systems, it's possible for our users to generate hundreds of millions of small files, which is difficult for our purge program to keep up with.  As a result, unnecessary files will continue to remain on the system.  More information on how to remove files with munlink can be found here.


We have also create script that will allow users to tar up directories in parallel, using MPIBash.  Users can download it here.

An example jobscript for using the parallel tar script:

#!/bin/bash --login

#SBATCH --nodes=1
#SBATCH --time=1:00:00
#SBATCH --export=NONE

module swap PrgEnv-cray PrgEnv-gnu
module load mpibash

export PMI_NO_PREINITIALIZE=1
export PMI_NO_FORK=1

srun --export=all -N 1 -n 12 mpi_tar.sh <directory_to_be_tarred>


The script first looks at the processor0 directory to figure out how many time steps there are in each directory, and then divides those up between the number of MPI processes you are using.  Each process tars up time steps in a processor* directory, and then a top level tar occurs and tars up all of the processor* directories, along with any other files or directory in the top-level openfoam case.

This will delete files after creating the tar files.


A few notes:

  • The script assumes that there are an equal number of time steps in each processor directory
  • This will only work on OpenFOAM directory structures, i.e.  it assumes there are multiple processor directories
  • A final tar file called  $directory.tar is created, where $directory is the name of the top-level directory you're archiving
  • There's no need to compress these with gzip...it will add a lot of time to the script and space isn't the issue with OpenFOAM files
  • At most users should use two nodes.  There will be little benefit in using more MPI ranks than that.  This is not a compute-intensive operation.

Contact the helpdesk if you have any questions about using the script.

The most recent versions of OpenFOAM: foundation's OpenFOAM-7 and ESI's OpenFOAM-v1912

The most recent versions of OpenFOAM are not currently installed on Pawsey systems. For running these, and for any other version of OpenFOAM not installed in our systems, we recommend the usage of Singularity containers with tools compiled with MPICH. MPICH is a requirement to be able to run MPI applications at Pawsey's Cray Supercomputers, please refer to Basics of Singularity - Using MPI.

For the most recent versions of OpenFOAM (and some others), Pawsey have prebuilt and tested Singularity containers. These containers are available in the following directory:

/group/singularity/pawseyRepository/OpenFOAM

At the time of writing, the following containers are available to users:

espinosa@magnus-1:/group/singularity/pawseyRepository/OpenFOAM> ls -lat
total 4607588
drwxrwsr-x  2 maali    pawsey0001       4096 Mar 23 15:40 .
drwxrwsr-x  3 maali    pawsey0001       4096 Mar  3 09:52 ..
-rwxr-xr-x+ 1 espinosa pawsey0001 1190842368 Feb 11 13:27 openfoam-v1912-pawsey.sif
-rwxr-xr-x+ 1 espinosa pawsey0001 1228185600 Feb 11 13:27 openfoam-7-pawsey.sif
-rwxr-xr-x+ 1 espinosa pawsey0001 1152237568 Feb 11 13:27 openfoam-5.x-pawsey.sif
-rwxr-xr-x+ 1 espinosa pawsey0001 1146875904 Nov 29 09:01 openfoam-v1812-pawsey.sif

Please follow the detailed instructions on how to use OpenFOAM containers at Pawsey for further information.

OpenFOAM versions not installed in our systems

For OpenFOAM versions not currently installed on Pawsey systems, users will need to build their own Singularity containers prepared with MPICH. Please refer to Basics of Singularity-Using MPI for an explanation on why MPICH.

Before building a new container, check if the version you are looking for is already available in the following directory:

/group/singularity/pawseyRepository/OpenFOAM

As shown in the previous section, Singularity containers with the most recent versions of OpenFOAM (and some others) have already been built and tested in our systems.

Please follow the detailed instructions on how to use OpenFOAM containers at Pawsey for further information.

For building a new container, please refer to how to build a new OpenFOAM container.

How to use OpenFOAM containers at Pawsey

As for every MPI application, every parallel solver needs to use the Supercomputer's MPICH (the one installed in the Supercomputer and not the MPI supplied within the container) as briefly explained in Basics of Singularity - Using MPI. The Singularity documentation (https://sylabs.io/docs/) refers to this mode of usage as the "hybrid mode". In order to run in hybrid mode, the MPI of the supercomputer and the MPI within the container need to be ABI compatible. MPICH is a requirement to be able to run MPI applications at Pawsey's Cray Supercomputers and is the only supported flavour to run containerised parallel tools for all our clusters.

The restrictions on the MPI flavour of the container implies that:

Native ESI or OpenFOAM-Foundation containers

Native developer's containers (from ESI or the OpenFOAM-Foundation) have been compiled with OpenMPI and are NOT supported to run paralllel MPI jobs on any of our clusters.

In the following sections we describe how to execute OpenFOAM containers at Pawsey Supercomputers.

0. Preparation

Once in a session on Zeus or Magnus, load the Singularity module:

user@zeus-1:~> module load singularity

Now, check the content in the directory to be used as a repository for the Singularity containers. The repository may be the one maintained by Pawsey:

user@zeus-1:~> theRepo=/group/singularity/pawseyRepository/OpenFOAM
user@zeus-1:~> ls $theRepo
openfoam-5.x-pawsey.sif  openfoam-7-pawsey.sif  openfoam-v1912-pawsey.sif

Or a project repository shared with the other members of your project:

user@zeus-1:~> theRepo=$MYGROUP/../singularity/groupRepository/OpenFOAM
user@zeus-1:~> ls $theRepo
openfoam-v1812-mickey.sif

Or your personal repository:

user@zeus-1:~> theRepo=$MYGROUP/singularity/myRepository/OpenFOAM
user@zeus-1:~> ls $theRepo
openfoam-7-mickey.sif

In the rest of this example, we'll use the openfoam-7-pawsey.sif within the pawseyRepository.

We recommend to use the following approach (or similar) to define the name of the container-image to use (specially if you are using or testing more than one container for your specific case):

user@zeus-1:~> theRepo=/group/singularity/pawseyRepository/OpenFOAM
user@zeus-1:~> theContainerBaseName=openfoam
user@zeus-1:~> theVersion=7
user@zeus-1:~> theProvider=pawsey
user@zeus-1:~> theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

Test if the container is responding correctly:

user@zeus-1:~> singularity exec $theImage icoFoam -help

You should see a message similar to this:

Usage: icoFoam [OPTIONS]
options:
  -case <dir>       specify alternate case directory, default is the cwd
  -fileHandler <handler>
                    override the fileHandler
  -hostRoots <(((host1 dir1) .. (hostN dirN))>
                    slave root directories (per host) for distributed running
  -libs <(lib1 .. libN)>
                    pre-load libraries
  -listFunctionObjects
                    List functionObjects
  -listRegisteredSwitches
                    List switches registered for run-time modification
  -listScalarBCs    List scalar field boundary conditions (fvPatchField<scalar>)
  -listSwitches     List switches declared in libraries but not set in
                    etc/controlDict
  -listUnsetSwitches
                    List switches declared in libraries but not set in
                    etc/controlDict
  -listVectorBCs    List vector field boundary conditions (fvPatchField<vector>)
  -noFunctionObjects
                    do not execute functionObjects
  -parallel         run in parallel
  -roots <(dir1 .. dirN)>
                    slave root directories for distributed running
  -srcDoc           display source code in browser
  -doc              display application documentation in browser
  -help             print the usage

Using: OpenFOAM-7 (see https://openfoam.org)
Build: 7-e84d903cab0c

1. Define the case to solve

Here, we describe how to use the "damBreak" tutorial provided by the OpenFOAM installed within the container. (Users with an already defined case to solve can skip the instructions in this section, although we recommend to read it to understand some features of container's usage).

The preparation of the tutorial can be done through both: an interactive or a non-interactive session. Here we explain how to do it non-interacively because job scripts are mostly prepared with non-interactive executions of the containers. When working non-interactively, a container executes the indicated command and exits immediately coming back to the shell prompt.

For example, to check the values of some of the OpenFOAM environment variables defined inside the container, the following commands may be executed:

user@zeus-1:~> singularity exec $theImage bash -c 'echo $WM_PROJECT_DIR'
/opt/OpenFOAM/OpenFOAM-7
user@zeus-1:~> singularity exec $theImage bash -c 'echo $FOAM_ETC'
/opt/OpenFOAM/OpenFOAM-7/etc
user@zeus-1:~> singularity exec $theImage bash -c 'echo $FOAM_TUTORIALS'
/opt/OpenFOAM/OpenFOAM-7/tutorials
  • bash -c 'command' executes the command between the apostrophes while in the container session. Thanks to the single apostrophes, the variables are evaluated inside the container and not in the host shell.

In order to create our own case from the tutorial, first find the family of tutorial cases for "damBreak":

user@zeus-1:~> singularity exec $theImage bash -c 'find $FOAM_TUTORIALS -type d -name "damBreak"'
/opt/OpenFOAM/OpenFOAM-7/tutorials/multiphase/interFoam/RAS/damBreak
/opt/OpenFOAM/OpenFOAM-7/tutorials/multiphase/interFoam/RAS/damBreak/damBreak
/opt/OpenFOAM/OpenFOAM-7/tutorials/multiphase/interFoam/laminar/damBreak
/opt/OpenFOAM/OpenFOAM-7/tutorials/multiphase/interFoam/laminar/damBreak/damBreak
/opt/OpenFOAM/OpenFOAM-7/tutorials/multiphase/interMixingFoam/laminar/damBreak

Once found, copy the desired case to /scratch:

user@zeus-1:~> cd $MYSCRATCH
user@zeus-1:~> mkdir -p ./OpenFOAM/$USER-$theVersion/run/example1
user@zeus-1:~> cd ./OpenFOAM/$USER-$theVersion/run/example1
user@zeus-1:~> singularity exec $theImage bash -c 'cp -r $FOAM_TUTORIALS/multiphase/interFoam/RAS/damBreak/damBreak .'
  • Note that: /group and /scratch are mounted by default for Singularity containers at Pawsey and containers are able to read/write on these file systems. Also, by default, Singularity mounts the directory from where it is called.

Now you can check of the content of the case directory:

user@zeus-1:~> ls damBreak
0  Allclean  Allrun  constant  system

(1.1 For the tutorial example, Increase the number of domains (processors) to be used in the simulation)

As we are demonstrating how to run a case in Pawsey supercomputers, we'll increase the numberOfSubdomains of the original tutorial to a more typical number. Edit the ./system/decomposeParDict file and define the use of 48 cores:

//numberOfSubdomains 4;
numberOfSubdomains 48;

method          simple;

simpleCoeffs
{
//    n               (2 2 1);
    n               (6 8 1);
        delta           0.001;
}

Please note that this is for demonstration purposes. In a real production case, the domain decomposition should assign around 100,000 computational cells per core, as explained in the best practices guidelines above.

2. Adapt the case to use Best Practices

All the cases to be ran at Pawsey supercomputers need to comply with the best practices guidelines. These guidelines are defined to avoid the problems of contention in Pawsey's shared file system.

For that, "cd" to the case directory

user@zeus-1:~> cd $MYSCRATCH/OpenFOAM/$USER-$theVersion/run/example1/damBreak

and edit your configuration dictionaries to comply with Pawsey best practices.

2.1 Set  to output of binary result files

Users should set the result files to be written in binary format. You can make some exceptions during the test and development phase of your project, but production runs should produce binary result files.

Edit the "/system/controlDict" dictionary file and set writeFormat to binary output:

//writeFormat     ascii;
writeFormat     binary;

2.2 If possible, set "purgeWrite" to a low number (different to 0)

This parameter defines the total number of time snapshots of results to be saved. Therefore, if this parameter is set to 10, then only the last 10 time snapshot directories will be kept. (When the parameter is set to "0", then all the saved snapshots will be kept.) Obviously, this is only useful whenever the results of the beginning of the simulation are not of interest to the user.

Edit the "system/controlDict" dictionary file and set purgeWrite to "10", for example:

//purgeWrite      0;
purgeWrite      10;

2.3 Never use "runTimeModifiable"

Users should not use the runTimeModifiable mode of OpenFOAM. This option may produce a large number of reads to the shared file system, which can produce contention and reduction of performance.

Edit the "system/controlDict" dictionary file and set runTimeModifiable to "no":

//runTimeModifiable yes;
runTimeModifiable no;

2.4 Use the combined option: -fileHandler collated -ioRanks '(0 G 2G ... mG)'

New versions of OpenFOAM (since OpenFOAM-6 and OpenFOAM-v1806) provide the "-fileHandler collated -ioRanks '(0 G 2G ... mG)'" combined option. This combined option allows for a huge reduction in the number of result directories and files generated (see explanation above). Therefore, if you are using a recent version of OpenFOAM, you should make use of it.

For recent OpenFOAM versions, containers maintained by Pawsey have already defined the first part of the combined option as default: "-fileHandler collated". This first part is the only one that can be pre-defined by default. If for your container does not have this set by default, then it needs to be set with one of the different methods described here. For example, edit the "system/controlDict" dictionary file and add:

OptimisationSwitches
{
    fileHandler collated;
}

ioRanks should be defined at execution time

Unfortunately, the second part of the combined option ("-ioRanks '(0 G 2G ... mG)'") cannot be pre-defined as a default or defined within the system/controlDict.

Therefore, users must define ioRanks at execution time following one the methods explained in this documentation (see explanation here). It's usage with containers will be exemplified in the following section with the use of the environment variable FOAM_IORANKS.

Simulations that rely only in the definition of the basic first part (-fileHandler collated) without the use of ioRanks may observe reduced performance, as all the results will be collated through a single processor. Therefore, remember to always define the second part of the combined option when a large number of processors (numberOfSubdomains) are to be used.

3. Execute containerised OpenFOAM tools and solvers

3.1 Serial pre-processing

Most of OpenFOAM pre-processing tools are serial. Here we execute several pre-processing tools as serial substeps defined in a job script. Use the following job script (preFOAM.sh) as an example:

#!/bin/bash -l 
#SBATCH --export=NONE
#SBATCH --time=00:15:00
#SBATCH --ntasks=1
#SBATCH --partition=workq
#SBATCH --clusters=zeus

#----------
# Load the necessary modules
module load singularity

#----------
# Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=7
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#----------
# Defining the case directory
myCase=$MYSCRATCH/OpenFOAM/$USER-$theVersion/run/example1/damBreak

#----------
# Defining the ioRanks for collating I/O
# groups of 24 as the intended parallel run will be executed in Magnus:
export FOAM_IORANKS='(0 24 48 72 96 120 144 168 192 216 240 264 288 312 336 360 384 408 432 456 480 504 528 552 576 600 624 648 672 696 720 744 768 792 816 840 864 888 912 936 960 984 1008 1032 1056 1080 1104 1128 1152 1176 1200)'

#----------
# Execute the pre-processing tools
echo "0: cd into the case directory"
cd $myCase
echo "I am in dir: $PWD"

echo "1: Creating the mesh:"
srun --export=all -n 1 -N 1 singularity exec $theImage blockMesh 2>&1 | tee log.blockMesh.$SLURM_JOBID

echo "2: Initialising fields:"
srun --export=all -n 1 -N 1 singularity exec $theImage setFields 2>&1 | tee log.setFields.$SLURM_JOBID

echo "3: Decomposing the case (ioRanks list will be picked up from FOAM_IORANKS environment variable):"
#A: When collated option is set by default or within system/controlDict:
srun -n 1 -N 1 singularity exec $theImage decomposePar 2>&1 | tee log.decomposePar.$SLURM_JOBID
#B: When collated option needs to be set explictly in the command line:
##srun -n 1 -N 1 singularity exec $theImage decomposePar -fileHandler collated 2>&1 | tee log.decomposePar.$SLURM_JOBID
  • The script is forced to run in zeus because serial pre- and post-processing of OpenFOAM cases should be executed in this cluster: –clusters=zeus
  • Singularity module needs to be loaded: module load singularity
  • For recent OpenFOAM versions, the collated/ioRanks combined option needs to be set for the decomposePar tool to properly decompose the domain. In this case, ioRanks is set through the environment variable FOAM_IORANKS which is exported towards the container by the srun command.

  • There is no problem with the large ioRanks list. If the value of any number in the list is greater than the real numberOfSubdomains (number of processors) set for the case in system/decomposeParDict, it will not generate problems and will simply not be taken into account (see explanation here).

To submit the job:

user@zeus-1:~> sbatch preFOAM.sh

As a result, the case is now prepared for executing the parallel solver. Notice that decomposed initial conditions are in the processorsN_X_Y directories:

user@zeus-1:~> ls $MYSCRATCH/OpenFOAM/$USER-7/run/example1/damBreak
0         Allrun    log.blockMesh.4190934     log.setFields.4190934  processors48_24-47
Allclean  constant  log.decomposePar.4190934  processors48_0-23      system
user@zeus-1:~> ls $MYSCRATCH/OpenFOAM/$USER-7/run/example1/damBreak/processors48_0-23/
0  constant

You should always check your slurm-<jobID>.out file created in the submission directory to confirm successful execution of the job script and for identifying errors. With the "2>&1 | tee" combination of commands, we are also creating the files log.<tool>.<jobID> within the case directory for the same purposes, but with separated outputs for each tool.

3.2 Executing the parallel solver

As mentioned previously, the OpenFOAM containers should be executed with the "hybrid mode" of Singularity. This mode makes use of the Supercomputer's MPICH installation binded to the MPICH installation inside the container. In Pawsey Supercomputers, binding is performed automatically when the Singularity module is loaded.

Use the following script (runFOAM.sh) as an example for executing the parallel solver:

#!/bin/bash -l 
#SBATCH --export=NONE
#SBATCH --time=00:15:00
#SBATCH --ntasks=48
#SBATCH --ntasks-per-node=24
#SBATCH --partition=workq
#SBATCH --clusters=magnus

#----------
# Load the necessary modules
module load singularity

#----------
# Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=7
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#----------
# Defining the case directory
myCase=$MYSCRATCH/OpenFOAM/$USER-$theVersion/run/example1/damBreak

#----------
# Defining the solver
of_solver=interFoam

#----------
# Defining the ioRanks for collating I/O
# groups of 24 as the intended parallel run will be executed in Magnus:
export FOAM_IORANKS='(0 24 48 72 96 120 144 168 192 216 240 264 288 312 336 360 384 408 432 456 480 504 528 552 576 600 624 648 672 696 720 744 768 792 816 840 864 888 912 936 960 984 1008 1032 1056 1080 1104 1128 1152 1176 1200)'

#----------
# Execute the parallel solver
echo "4: cd into the case directory"
cd $myCase
echo "I am in dir: $PWD"

echo "5: Executing the case in parallel (ioRanks list will be picked up from FOAM_IORANKS environment variable):"
#A: When collated option is set by default or within system/controlDict:
srun -n $SLURM_NTASKS -N $SLURM_JOB_NUM_NODES singularity exec $theImage $of_solver -parallel 2>&1 | tee log.$of_solver.$SLURM_JOBID
#B: When collated option needs to be set explictly in the command line:
#srun -n $SLURM_NTASKS -N $SLURM_JOB_NUM_NODES singularity exec $theImage $of_solver -parallel -fileHandler collated 2>&1 | tee log.$of_solver.$SLURM_JOBID
  • The script is forced to run in magnus with --ntasks-per-node=24
  • Singularity module needs to be loaded: module load singularity
  • For recent OpenFOAM versions, the collated/ioRanks combined option needs to be set for the interFoam solver to output the results properly. In this case, ioRanks is set through the environment variable FOAM_IORANKS which is exported towards the container by the srun command.

  • There is no problem with the large ioRanks list. If the value of any number in the list is greater than the real numberOfSubdomains (number of processors) set for the case in system/decomposeParDict, it will not generate problems and will simply not be taken into account (see explanation here).

To submit the job (job will run in Magnus):

user@zeus-1:~> sbatch runFOAM.sh

After the job finishes, the results should be in the processors48_0-23, processors48_24-47 directories:

user@zeus-1:~> ls $MYSCRATCH/OpenFOAM/$USER-7/run/example1/damBreak/processors48_0-23/
0  0.55  0.6  0.65  0.7  0.75  0.8  0.85  0.9  0.95  1  constant

You should always check your slurm-<jobID>.out file created in the submission directory to confirm successful execution of the job script and for identifying errors. With the "2>&1 | tee" combination of commands, we are also creating the files log.<tool>.<jobID> within the case directory for the same purposes, but with separated outputs for each tool.

3.3 Serial post-processing

The reconstruction of the case is again a serial task. For this, you can use the following script (postFOAM.sh) as an example:

#!/bin/bash -l
#SBATCH --export=NONE
#SBATCH --time=00:15:00
#SBATCH --ntasks=1
#SBATCH --partition=workq
#SBATCH --clusters=zeus

#----------
# Load the necessary modules
module load singularity

#----------
# Defining the container to be used
theRepo=/group/singularity/pawseyRepository/OpenFOAM
theContainerBaseName=openfoam
theVersion=7
theProvider=pawsey
theImage=$theRepo/$theContainerBaseName-$theVersion-$theProvider.sif

#----------
# Defining the case directory
myCase=$MYSCRATCH/OpenFOAM/$USER-$theVersion/run/example1/damBreak

#----------
# Defining the ioRanks for collating I/O
# groups of 24 as the intended parallel run will be executed in Magnus:
export FOAM_IORANKS='(0 24 48 72 96 120 144 168 192 216 240 264 288 312 336 360 384 408 432 456 480 504 528 552 576 600 624 648 672 696 720 744 768 792 816 840 864 888 912 936 960 984 1008 1032 1056 1080 1104 1128 1152 1176 1200)'

#----------
# Execute the pre-processing tools
echo "6: cd into the case directory"
cd $myCase
echo "I am in dir: $PWD"

echo "7: Reconstructing the case (ioRanks list will be picked up from FOAM_IORANKS environment variable):"
#A: When collated option is set by default or within system/controlDict:
srun -n 1 -N 1 singularity exec $theImage reconstructPar -latestTime 2>&1 | tee log.reconstructPar.$SLURM_JOBID
#B: When collated option needs to be set explictly in the command line:
##srun -n 1 -N 1 singularity exec $theImage reconstructPar -latestTime -fileHandler collated 2>&1 | tee log.reconstructPar.$SLURM_JOBID
  • The script is forced to run in zeus because serial pre- and post-processing of OpenFOAM cases should be executed in this cluster: –clusters=zeus
  • Singularity module needs to be loaded: module load singularity
  • For recent OpenFOAM versions, the collated/ioRanks combined option needs to be set for the reconstructPar tool to properly reconstruct the case. In this case, ioRanks is set through the environment variable FOAM_IORANKS which is exported towards the container by the srun command.

  • There is no problem with the large ioRanks list. If the value of any number in the list is greater than the real numberOfSubdomains (number of processors) set for the case in system/decomposeParDict, it will not generate problems and will simply not be taken into account (see explanation here).

To submit the job:

user@zeus-1:~> sbatch postFOAM.sh

The reconstructed directory "1" should be available in the root of the case directory:

user@zeus-1:~> ls $MYSCRATCH/OpenFOAM/$USER-7/run/example1/damBreak
0  Allclean  constant  log.blockMesh.4199306     log.interFoam.4773027       log.setFields.4199306  processors48_24-47
1  Allrun    core      log.decomposePar.4199306  log.reconstructPar.4199472  processors48_0-23      system
user@zeus-1:~> ls $MYSCRATCH/OpenFOAM/$USER-7/run/example1/damBreak/1
alphaPhi0.water  alpha.water  epsilon  k  nut  p  phi  p_rgh  U  uniform

You should always check your slurm-<jobID>.out file created in the submission directory to confirm successful execution of the job script and for identifying errors. With the "2>&1 | tee" combination of commands, we are also creating the files log.<tool>.<jobID> within the case directory for the same purposes, but with separated outputs for each tool.

Training workshop for OpenFOAM containers

As complementary (and very useful) information, we have prepared a workshop covering several topics related to the use of OpenFOAM containers at Pawsey. The training workshop for OpenFOAM containers at Pawsey (https://pawseysc.github.io/containers-openfoam-workshop/) covers the following topics:

  1. Basic information for OpenFOAM containers at Pawsey
  2. Executing the full workflow of a case
  3. Compile and execute user's own tools
  4. Building an OpenFOAM container with MPICH
  5. Use OverlayFS to reduce the number of result files
  6. Advanced scripts for postprocessing with OverlayFS

This is a great source of additional information and we encourage OpenFOAM users to read the tutorials and practise the exercises.

  • No labels