Applications for this scheme are currently closed for resources in 2023.
Setonix availability in January 2023
Setonix will experience downtime in the beginning of January 2023 while final acceptance testing is carried out.
The National Computational Merit Allocation Scheme (NCMAS) is Australia’s premier meritorious allocation scheme, spanning both national peak facilities as well as specialised compute facilities across the nation.
The NCMAS is open to the Australian research community, providing significant amounts of compute time for meritorious research projects.
The NCMAS is administered by the NCMAS secretariat.
Please find below the link to the NCMAS application portal. Please refer to the official NCMAS communication about the opening and closing dates of the call.
Further information is available at https://ncmas.nci.org.au.
Available Resources in 2023
The NCMAS is one of the Merit Allocation Schemes available on Setonix. Researchers will experience a substantial increase in computational resources available in the NCMAS scheme due to the delivery of Setonix Phase 2 in 2023. Researchers can apply for allocations on Setonix CPU and Setonix GPU.
Resources available and minimum allocation sizes are presented in table 1.
Table 1. Available resources and minimum request size
National Computational Merit Allocation Scheme
|Scheme total capacity|
455M Service Units Total:
|Minimum request size||1M Service Units|
There is no maximum limit to the amount of time that can be requested. However, partial allocations may be awarded depending on the availability and demand for allocations within the scheme.
Note that 1M core hours in a year is approximately the equivalent of using a single Setonix CPU node. Applications for such small allocations must specify why access to a supercomputer is necessary for the research, and based on the scoring criteria below such uses of the supercomputer are unlikely to be competitive against other applications that demonstrate they need the expensive interconnect. Nimbus Documentation is better suited to single-node applications and has a lightweight application process.
Other non-Pawsey resources are available under the NCMAS including NCI's Gadi supercomputer.
How to estimate Service Units request for Setonix-GPU?
Researchers planning their migration from NVIDIA-based GPU systems like Pawsey’s Topaz and NCI’s Gadi to AMD-based Setonix-GPU should use the following example strategy to calculate their Service Units request.
- Simulation walltime on a single NVIDIA V100 GPU: 1h
- Safe estimate for Service Units usage on a single Setonix’s AMD MI250X GPU: 1h * 1/2 * 128 = 64 Service Units
Please see: https://www.amd.com/en/graphics/server-accelerators-benchmarks
Setonix-GPU migration pathway
The Setonix’s AMD MI250X GPUs have a very specific migration pathway related to CUDA to HIP and OpenACC to OpenMP conversions. Pawsey is working closely with research groups within PaCER project (https://pawsey.org.au/pacer/) and with vendors to further extend the list of supported codes.
Please see: https://www.amd.com/en/technologies/infinity-hub
With Setonix, Pawsey is moving from an exclusive node usage to a proportional node usage accounting model. While the Service Unit (SU) is still mapped to the hourly usage of CPU cores, users are not charged for whole nodes irrespective of whether they are been fully utilised. With the proportional node usage accounting model, users are charged only for the portion of a node they requested.
Each CPU compute node of Setonix can run multiple jobs in parallel, submitted by a single user or many users, from any project. Sometimes this configuration is called shared access.
A project that has entirely consumed its service units (SUs) for a given quarter of the year will run its jobs in low priority mode, called extra, for that time period. Furthermore, if its service unit consumption for that same quarter hits the 150% usage mark, users of that project will not be able to run any more jobs for that quarter.
Pawsey accounting model bases the GPU charging rate on energy consumption. Such approach, designed for Setonix, has a lot of advantages compared to other models, introducing carbon footprint as a primary driver in determining the allocation of computational workflow on heterogeneous resources.
Pawsey and NCI centres are using slightly different accounting models. Researchers applying for allocations on Setonix and Gadi should refer to Table 2 when calculating their allocation requests.
Table 2. Setonix and Gadi service unit models
|Resources used||Service Units|
CPU: 48 Intel Cascade Lake cores per node
GPU: 4 Nvidia V100 GPUs per node
CPU: 128 AMD Milan cores per node
GPU: 4 AMD MI250X GPUs per node
|1 CPU core / hour||2||1|
|1 CPU / hour||48||64|
|1 CPU node / hour||96||128|
|1 GPU / hour||36*||128|
|1 GPU node / hour||144*||512|
* calculated based on https://opus.nci.org.au/display/Help/2.2+Job+Cost+Examples for gpuvolta queue
Criterion 1: Project quality and innovation
- Significance of the research
- Originality and innovative nature of the computational framework
- Advancement of knowledge through the goals of the proposed research
- Potential for the research to contribute to Australian science, research and innovation priorities
Criterion 2: Investigator records
- Research record and performance relative to opportunity (publications, research funding, recognition and esteem metrics)
Criterion 3: Computational feasibility
- Adequacy of the time commitment of investigators to undertake the research and utilise the resources successfully
- Suitability of the system to support the research, and appropriate and efficient use of the system
- Capacity to realise the goals of the project within the resources request
- Appropriate track record in the use of high-performance computing systems, relative to the scale of the resources requested
Criterion 4: Benefit and impact
- The ability of the project to generate impactful outcomes and produce innovative economic, environmental and social benefits to Australia and the international community
Application Form and Process
Applications to the NCMAS are via online form at the NCMAS website, https://ncmas.nci.org.au.
No Partner Top-up Allocations
There will be no Pawsey Partner top-up allocations starting from 2023 allocation round. Researchers can apply to both NCMAS and Pawsey Partner Scheme subject to the eligibility and conditions of these schemes.
- Allocation Schemes and Eligibility
- Completing an Application for Allocation
- Accessing Pawsey Supercomputers
- Cristian Di Pietrantonio, Christopher Harris, Maciej Cytowski, "Energy-based Accounting Model for Heterogeneous Supercomputers"