Magnus is a Cray XC40 supercomputer. That is, it is a massively parallel architecture of many individual nodes that are connected by a high speed network. Each node is based on the x86-64 architecture, which is popular in desktop PCs and enterprise servers.
Magnus is a homogeneous cluster which means there is just one type of processor across all compute nodes; the Intel Xeon E5-2690 v3 (Haswell). Each node has two E5-2690 v3 processors and each E5-2690 v3 has 12 cores. These can be referred to as “dual 12-core” nodes. In the simplest of terms Magnus can be considered to have twenty four cores per node providing a total of 35,712 cores across the 1488 nodes. Each node has 64GB of memory.
Connecting all the nodes together is the HPC optimized Aries interconnect, that utilizes the Dragonfly network topology to enable parallel codes using MPI – Message Passing Interface, to communicate between the compute nodes. Each compute blade has a single Aries ASIC, which provides about 72Gbits/sec for each of the 4 nodes on the blade. The Aries/Dragonfly interconnect has been designed for low latency and efficient small message transfer.
Storage on Magnus is provided by the Lustre parallel file system. Lustre can use multiple servers to store metadata and data, which can improve throughput significantly for large data files. The Lustre filesystem operates over the Infiniband network.
Magnus has two login nodes. The generic hostname magnus.pawsey.org.au distributes users onto either login node via round-robin DNS. To run jobs on the compute nodes, submit jobs from the login nodes. Compute nodes cannot be accessed from the internet.
Galaxy is a Cray XC30 supercomputer. That is, it is a massively parallel architecture of many individual nodes that are connected by a high-speed network. Each node is based on the x86-64 architecture, which is popular in desktop PCs and enterprise servers: a small number of nodes also have a GPU accelerator to support, amongst others, CUDA-enabled applications.
Galaxy consists of three cabinets that contain 134 compute blades, each of which has four nodes. The majority of nodes (472, in 118 blades) support two 10-core Intel Xeon E5-2690V2 'Ivy Bridge' processors operating at 3.00 GHz with 64GB of memory, for a total of 9,440 cores delivering around 200 TFLOPS of compute power. There are also 64 nodes that each support an NVIDIA K20X 'Kepler' GPU, alongside an 8-core Intel Xeon E5-2690 'Sandy Bridge' host processor at 2.6 GHz, with 32GB of (host) memory.
Connecting all the nodes together is the HPC-optimized Cray Aries interconnect, which utilizes the Dragonfly network topology to enable parallel codes using MPI – Message Passing Interface, to communicate between the compute nodes. Each compute blade has a single Aries ASIC, which provides about 72Gbits/sec for each of the 4 nodes on the blade. The Aries interconnect and Dragonfly topology has been designed for low latency and efficient small message transfer.
Galaxy has a 1.3 Petabyte Lustre scratch file system, provided by a Cray Sonexion 1600 appliance storage system and connected via FDR Infiniband.
Galaxy has two login nodes. The generic hostname galaxy.pawsey.org.au distributes users onto either login node via round-robin DNS. To run jobs on the compute nodes, submit jobs from the login nodes. Compute nodes cannot be accessed from the internet.
Zeus is an SGI Linux cluster. Zeus is primarily used for pre- and post-processing of data, large shared memory computations and remote visualization work. Zeus is a heterogeneous cluster with 30 nodes in various configurations. Zeus shares the same /home, /scratch and /group file systems with Magnus. Users ssh to zeus.pawsey.org.au to access all the 30 nodes in Zeus.
In Zeus, there is one large-memory node called Zythos, which is an SGI UV2000 system. Zythos consists of 24 UV blades, and they appear as a single system with huge shared memory of 6TB. 20 of the blades each contain two 6-core Intel Xeon E5-4610 CPUs, and the other 4 blades each contain one 6-core Intel Xeon E5-4610 CPU and one Nvidia Tesla K20 GPU. Multiple users are permitted to run on Zythos at the same time.
Apart from Zythos, Zeus’s other 29 nodes all have two 8-core Intel Xeon E5-2670 CPUs. 27 of these nodes also feature GPUs, with 20 of them each containing one Nvidia Quadro K5000 GPU and the other 7 nodes each containing one Nvidia Tesla K20 GPU. On these 27 GPU-featuring nodes, only one user is permitted to run at a time on each node.
Time on Zythos is allocated through the Pawsey Director’s Share. For Zeus's other nodes, access is automatically granted to anyone who is a user in an active project on either Magnus, Galaxy or Zythos.
Zeus is part of the portfolio of Pawsey Project resources: it resides in the Pawsey Centre and is closely integrated with other Pawsey Centre infrastructure including the Magnus supercomputer, the RDSI Collection Development node, and the Hierarchical Storage Management system, allowing a diverse range of workflows to be undertaken.
Zythos, a latest-generation SGI UV2000 system, is a large shared-memory machine targeted towards memory-intensive jobs. Zythos consists of 24 UV blades. Twenty of the blades each contain two hex-core Intel “Sandy Bridge” processors and 256GB RAM, and the remaining four each contain a single hex-core Intel processor, an NVIDIA Kepler K20 GPU, and 256 GB RAM. Altogether the machine boasts 264 CPU-cores, 4 GPUs, and a total of 6TB RAM. Zythos is a partition within the Zeus cluster.
Zythos is primarily for pre- and post-processing of data associated with Magnus or Galaxy. Due to its large amount of shared memory, Zythos is also well suited to scientific problems that cannot easily make use of distributed memory. Zythos provides researchers with an opportunity to undertake novel and previously impractical computational, data-analysis, and visualisation workflows. Time on Zythos is allocated through the Pawsey Supercomputing Centre Director’s Share.