This scheme is currently closed for applications for resources in 2023 allocation round.
Setonix will experience downtime in the beginning of January 2023 while final acceptance testing is carried out.
The Pawsey Partner Scheme supports the Pawsey partner institutions: CSIRO, Curtin University, Edith Cowan University, Murdoch University and the University of Western Australia. The scheme provides significant amounts of compute time for meritorious projects led by researchers from these institutions.
Staff from Western Australian Government Agencies who are conducting research or development are treated as belonging to a partner institution for the purpose of being eligible to access Pawsey services under the Partner Share. Such staff are eligible to be Lead Chief Investigator of Partner Share projects and contribute to the 50% FTE criteria.
The Pawsey Partner Scheme Call is published once a year in August/September for allocations in the coming year.
Available resources in 2023
The Pawsey Partner Scheme is one of the Merit Allocation Schemes available on Setonix. The computational resources available to researchers through the Pawsey Partner Scheme will increase substantially with the delivery of Setonix Phase 2 in 2023. Researchers can apply for allocations on Setonix CPU and Setonix GPU.
Resources available and minimum allocation sizes are presented in table 1.
Table 1. Available resources and minimum request size
|Pawsey Partner Merit Allocation Scheme||Scheme total capacity|
540M Service Units
|Minimum request size||1M Service Units|
An additional 5,000,000 SUs will be set aside that can be applied for throughout the year by new project leaders that have recently become eligible for the scheme. The set-aside amount will reduce pro-rata through the year.
There is no maximum limit to the amount of time that can be requested. However, partial allocations may be awarded depending on the availability and demand for allocations within the scheme.
Note that 1M core hours in a year is approximately the equivalent of using a single Setonix CPU node. Applications for such a small allocation must specify why access to a supercomputer is necessary for the research. Based on the scoring criteria below such uses of the supercomputer are unlikely to be competitive against other applications that demonstrate they need the expensive interconnect. Cloud Documentation is better suited to single-node applications and has a lightweight application process.
With Setonix, Pawsey is moving from an exclusive node usage to a proportional node usage accounting model. While the Service Unit (SU) is still mapped to the hourly usage of CPU cores, users are not charged for whole nodes irrespective of whether they are been fully utilised. With the proportional node usage accounting model, users are charged only for the portion of a node they requested.
Each CPU compute node of Setonix can run multiple jobs in parallel, submitted by a single user or many users, from any project. Sometimes this configuration is called shared access.
A project that has entirely consumed its service units (SUs) for a given quarter of the year will run its jobs in low priority mode, called extra, for that time period. Furthermore, if its service unit consumption for that same quarter hits the 150% usage mark, users of that project will not be able to run any more jobs for that quarter.
Pawsey accounting model bases the GPU charging rate on energy consumption. Such approach, designed for Setonix, has a lot of advantages compared to other models, introducing carbon footprint as a primary driver in determining the allocation of computational workflow on heterogeneous resources.
Pawsey and NCI centres are using slightly different accounting models. Researchers applying for allocations on Setonix should refer to Table 2 when calculating their allocation requests.
Table 2. Setonix service unit models
|Resources used||Service Units|
CPU: 128 AMD Milan cores per node
GPU: 4 AMD MI250X GPUs per node
|1 CPU core / hour||1|
|1 CPU / hour||64|
|1 CPU node / hour||128|
|1 GPU / hour||128|
|1 GPU node / hour||512|
How to estimate Service Units request for Setonix-GPU?
Researchers planning their migration from NVIDIA-based GPU systems like Pawsey’s Topaz and NCI’s Gadi to AMD-based Setonix-GPU should use the following example strategy to calculate their Service Units request.
- Simulation walltime on a single NVIDIA V100 GPU: 1h
- Safe estimate for Service Units usage on a single Setonix’s AMD MI250X GPU: 1h * 1/2 * 128 = 64 Service Units
Please see: https://www.amd.com/en/graphics/server-accelerators-benchmarks
Setonix-GPU migration pathway
The Setonix’s AMD MI250X GPUs have a very specific migration pathway related to CUDA to HIP and OpenACC to OpenMP conversions. Pawsey is working closely with research groups within PaCER project (https://pawsey.org.au/pacer/) and with vendors to further extend the list of supported codes.
Please see: https://www.amd.com/en/technologies/infinity-hub
- The Lead Chief Investigator (Lead CI) must hold a substantive position at a Pawsey Partner institution. A researcher undertaking a higher degree by research, or holding only an adjunct position, is not eligible to be a Lead CI.
- A researcher can be listed as Lead Chief Investigator or Chief Investigator on one application only. However, a researcher can appear as an “Other Project Participant” on multiple applications. This is to ensure that their track record is counted for one application only while encouraging collaborations between groups in different disciplines and universities.
- The applicant (the person who completes the application form) cannot be a student. Ideally, the Lead CI is the applicant.
- To be eligible for the Pawsey Partner scheme, at least 50% of the total FTE must be from among the Pawsey Partner institutions.
- Chief Investigators (CIs) of the project team may be staff at academic institutions or research organisations, including those located internationally as well as within Australia.
- Lead CI and CIs can work collaboratively on the proposal and must provide "Research Opportunity, Performance Evidence, Research Context and Experience in HPC" document, which contributes directly to the assessment score.
- Other Project Participants may be staff or students at academic institutions or research organisations, including those located internationally as well as within Australia.
- The use of Pawsey Project Infrastructure is conditional on complying with relevant laws and export controls, including the Australian Defence Trade Controls Act, United National Security Council (UNSC) sanctions regimes and the Australian autonomous sanctions regimes, and U.S. Export Controls.
Applications are assessed against the following criteria. To be competitive, particularly for larger requests, it is crucial that applications are of a high standard and address all criteria. The research criteria have a combined weighting of 70%. The technical criteria have a combined weighting of 30%.
Note that the research criteria are flexible, and it is up to the applicant to make a strong case based on their track record and the research project. For example, academic researchers will typically highlight their publication record and the scientific benefits of their research, whereas academics partnering with industry may choose to focus on industry grants and transformational benefits to the company of their work. Patents may also form part of the track record and/or project outputs.
Project Quality and Innovation (research criteria, 30%)
- Significance of the research
- Originality and innovative nature of the computational framework
- Advancement of knowledge through the goals of the proposed research
Investigators (research criteria, 15%)
- Research record and performance
Appropriateness of Request (research criteria, 15%)
- Justification of the size of the request
- Usage relative to previous allocations
- Current request relative to the previous usage
- Capacity to complete the project successfully within the original request
Benefit and Impact (research criteria, 10%)
- Ability of the project to generate impactful outcomes and produce innovative economic, environmental, and social benefits
Suitability (technical criteria, 15%)
- Method of parallelisation, including job packing, OpenMP, MPI and data/workflow parallelism
- Footprint of workflow, including wall times, memory, storage, and transfers
Scalability (technical criteria, 15%)
- Scale of workflow in terms of size and number of jobs
No Partner top-up allocations
There will be no Pawsey Partner top-up allocations starting from 2023 allocation round. Researchers can apply to both NCMAS and Pawsey Partner Scheme subject to the eligibility and conditions of these schemes.
Applications for the Pawsey Partner scheme outside of the official call may be considered in exceptional cases. These applications will be subject to the normal review criteria by the Pawsey Partner panel. Allocations will terminate with those of the corresponding main call. There is a cap of 2,000,000 Service Units per project, which is reduced pro-rata based on the months remaining in the year.
An example of an exceptional case is a newly recruited Project Leader into a Pawsey Partner institution. Another example is if a significant industry collaboration is initiated after the primary call for applications. The decision whether to accept an out of session application rests with the Chair of the Pawsey Partner panel.
Forgetting to apply or not being aware of the call for applications do not constitute exceptional cases.
To maintain the integrity of the independent merit allocation committee, the Pawsey Supercomputing Centre will not make adjustments to allocations made by the committee. This also includes requests for extra time for a project throughout the year.
Appeals can be made against procedural issues, but not against decisions by the reviewers, review panels or Pawsey Executive Director. Appeals should be emailed to firstname.lastname@example.org.
Application form and process
Applications for the Pawsey Partner scheme must be submitted online via the application portal, after the call commences and before the call closes.
To ensure Pawsey has the information needed to properly assess a proposal, a researcher should ensure that they:
- meet the criteria of the call to which they are applying
- read the call text carefully and aim to address all points of consideration
- provide clear and complete information
More specifically, the application form requires the following information:
- Project summary
- Research and computational classification
- List of participants (Lead Chief Investigator, Chief Investigators, Other Project Participants)
- Project Team - Chief Investigators
- Research record
- List of publications for last five years
- Research funding information for the last five years
- Research Proposal
- Project description
- Computational methodology
- Applications, tools and libraries
- Resource and storage request - calculate service units, memory and data requirements
- Progress report (if continuing)
A project should have the following defined:
- time-frame (specified start and end point)
- scientific goals
- expected research impact
- technical approach (including, software, hardware, tools and techniques)
- resources (people effort and expertise)
- cost (compute time and short-term storage requirements)
- When evaluating a proposal, the selection panel considers the proposer’s performance in any previous compute-time awards they may have received – in terms of fulfillment of goals and use of allocated resources.
- If a researcher has under-utilised a previous merit allocation, they should include an explanation of the reason for not using the allocated time.
- Allocation of compute time on Pawsey resources remains the discretion of the Pawsey Executive Director; their decision regarding the assessment of a proposal or the allocation of compute time is final.
- Other National and International Resources
- Cloud Documentation
- Allocation Schemes and Eligibility
- Completing an Application for Allocation
- Accessing Pawsey Supercomputers
- Cristian Di Pietrantonio, Christopher Harris, Maciej Cytowski, "Energy-based Accounting Model for Heterogeneous Supercomputers"