The Weather Research and Forecasting (WRF) Model is a next-generation mesoscale numerical weather prediction system designed for both atmospheric research and operational forecasting applications. It features two dynamical cores, a data assimilation system, and a software architecture supporting parallel computation and system extensibility. The model serves a wide range of meteorological applications across scales from tens of meters to thousands of kilometres.
For more information, see:
Before you begin
Researchers using WRF on Pawsey supercomputing systems typically need to build their own custom version to support their simulations. For this reason, WRF is not provided as a module by Pawsey staff. However, the WRF dependencies are installed on the system.
Registration with WRF is also required to before downloading, which should be completed before following the instructions below.
Installing WRF using an interactive session
The following instructions provide guidance to compile WRF on Setonix.
Step 1: Start an interactive session on the work partition
Step 2: Change directory to the /software filesystem
We recommend building and compiling WRF on the /software filesystem, so it will not be purged by the 30 day purge policy on /scratch.
Move to your software directory using the
$MYSOFTWARE environment variable shown below, and create and enter any additional directories if desired.
$ cd $MYSOFTWARE
Step 3: Download WRF
Check the for the URL to the latest version on the releases page, this will likely be the file ending in tar.gz in the Assets box at the bottom of the latest release.
Use the wget utility to download the file on the debug node, as it is a relatively small file that should only takes a few seconds.
Step 3: Extract WRF
Expand the WRF tarball using the
$ tar zxvf v4.4.tar.gz
Step 4: Enter the WRF directory
cd command to enter the WRFV4.4 directory:
$ cd WRFV4.4
Step 5: Prepare the Environment
Swap to the PrgEnv-cray programming environment, load the modules containing dependencies needed by WRF, and set up the
NETCDF environment variable.
Step 6: Configure WRF
Run the WRF configuration script:
Select the option that contains the Cray compiler and the corresponding type of parallelism desired.
For WRF 4.4 these options are:
46. (dmpar) CRAY CCE (ftn $(NOOMP)/cc): Cray XE and XCfor MPI support, or
47. (smpar_dmpar) CRAY CCE (ftn $(NOOMP)/cc): Cray XE and XCfor MPI and OpenMP support.
dmpar configuration (option 46) is generally used over the hybrid approach (option 47).
Select the level of nesting as desired, which in most cases will the default option corresponding to basic nesting.
Step 7: Compile WRF
Run the relevant WRF compilation, which may take one to two hours:
Installing WRF using an Slurm batch script
WRF can also be installed using a Slurm batch script instead of an interactive session.
Note that the options to the interactive configure script must be provided.
The following is an example for Option 46 (described in the previous section) and the default nesting.
Using WRF on Setonix
To use the WRF installation on Setonix, we recommend:
- Copying your WRF installation to a directory for the simulation on /scratch
- Copy necessary input data files as needed to the directory
- Configure your simulation
- Prepare a Slurm batch script to launch the job, including the same environment commands used to compile WRF
- Submit your job to the scheduler
The following is an example jobscript for running an MPI-enabled (dmpar) version of WRF on 2 nodes, or 256 cores:
WRF Reference Data
The WPS Geog dataset has been placed in
/scratch/references/wrf/wps_geog to avoid the need for repeated staging of data commonly used by WRF users.
For more information, see WPS Geographical Static Data webpage (external link).
WPS can be compiled in a similar manner to WRF.
Solutions to Common Issues
WRF simulation crashes due to numerical accuracy in the noah_mp module.
WRF contains internal checks on the ongoing accuracy of the simulation, and will halt the simulation if accuracy conditions are not met.
In practice, this looks like the following in one of the many rsl.out.* files created when running a simulation:
In this example there is a known issue when using the noah_mp land surface model in WRF when compiled using the Cray Fortran compiler that causes numerical instabilities and model crash. This has been linked to compiler optimisation levels and a work-around is to compile the noah_mp module with -O1 optimisation, but the rest of WRF with -O3 optimisation. To do this, you first need to compile WRF as normal.
Then you need to manually delete the compiled noah_mp files, re-compile these with -O1, and then run ./compile em_real again. Note the compile example below may have different paths depending on the location of WRF.
WRF 4.4 bug in phys/module_ra_clWRF_support.F
Be aware that the current 4.4 version of WRF contains a bug that will cause wrf.exe to crash:
This requires code changes in the phys/module_ra_clWRF_support.F file around line 171, specifically the lines:
need to be changed to:
It is expected that versions newer than 4.4 will address this bug.