Introduction
The Astonomy Filesystem /astro is a lustre filesystem provided for the scratch space needs of the MWA group who perform computations on the Garrawarla cluster. It is an SGI/HPE provided cluster of nodes backed by DDN storage. The system currently contains 2 Metadata servers (MDS) with 2 Metadata targets (MDT). It has 4 Object Store servers (OSS) for storing data and they have 48 Object Store targets (OST). This gives approximately 2.7 PB of usable storage. It has a possible read and write speed of over 10GB/s and Pawsey staff have been easily getting 7-8GB/s when using only the four copyq nodes to transfer data around using dcp. The single threaded IO speed is greater than on /scratch2 due to the updated version of Lustre.
The expandability of lustre means that the filesystem can be expanded, without downtime, by adding more OSS's and OST Disk behind them in groups of 2 (for high availability).
Location
The Astronomy filesystem is mounted on all Garrawarla, Magnus, Zeus, Topaz and Galaxy nodes and data mover nodes as /astro. The top level directory has directories for all the areas that /astro has:
garrawarla-1:/astro # ls -l /astro/ total 20 drwxrws--- 36 root mwaeor 4096 Oct 28 09:52 mwaeor drwxrws--- 12 root mwaops 4096 Sep 30 16:31 mwaops drwxrws--- 47 root mwasci 4096 Dec 11 16:56 mwasci drwxrwsr-x 57 root mwavcs 4096 Dec 16 16:46 mwavcs drwxrws--- 27 root pawsey0001 4096 Dec 17 12:22 pawsey0001
The pawsey0001 directory is for Pawsey testing of the system and can be set up in different ways as needed, it will not often be used.
Quotas
At the time of writing MWA have requested that mwaeor, mwavcs, mwaops and mwasci are assigned 370TB, 580TB, 20TB and 600T respectively.
To check the current quota use:
lfs quota -g projectcode /astro
Usage
To check usage you can use the normal unix df command to check the entire filesystem.
galaxy-1:/astro # lfs df -h /astro/ UUID bytes Used Available Use% Mounted on astrofs-MDT0000_UUID 542.1G 26.0G 479.4G 6% /astro[MDT:0] astrofs-MDT0001_UUID 542.1G 32.7G 472.7G 7% /astro[MDT:1] astrofs-OST0000_UUID 57.7T 26.6T 28.1T 49% /astro[OST:0] astrofs-OST0001_UUID 57.7T 26.4T 28.4T 49% /astro[OST:1] astrofs-OST0002_UUID 57.7T 25.9T 28.8T 48% /astro[OST:2] astrofs-OST0003_UUID 57.7T 28.6T 26.2T 53% /astro[OST:3] astrofs-OST0004_UUID 57.7T 26.9T 27.9T 50% /astro[OST:4] astrofs-OST0005_UUID 57.7T 26.6T 28.2T 49% /astro[OST:5] astrofs-OST0006_UUID 57.7T 26.3T 28.4T 49% /astro[OST:6] astrofs-OST0007_UUID 57.7T 26.8T 28.0T 49% /astro[OST:7] astrofs-OST0008_UUID 57.7T 26.8T 27.9T 50% /astro[OST:8] astrofs-OST0009_UUID 57.7T 26.3T 28.5T 48% /astro[OST:9] astrofs-OST000a_UUID 57.7T 26.6T 28.2T 49% /astro[OST:10] astrofs-OST000b_UUID 57.7T 26.8T 28.0T 49% /astro[OST:11] astrofs-OST000c_UUID 57.7T 25.6T 29.2T 47% /astro[OST:12] astrofs-OST000d_UUID 57.7T 27.1T 27.6T 50% /astro[OST:13] astrofs-OST000e_UUID 57.7T 27.0T 27.8T 50% /astro[OST:14] astrofs-OST000f_UUID 57.7T 26.5T 28.3T 49% /astro[OST:15] astrofs-OST0010_UUID 57.7T 37.3T 17.5T 69% /astro[OST:16] astrofs-OST0011_UUID 57.7T 38.1T 16.7T 70% /astro[OST:17] astrofs-OST0012_UUID 57.7T 37.5T 17.3T 69% /astro[OST:18] astrofs-OST0013_UUID 57.7T 38.0T 16.7T 70% /astro[OST:19] astrofs-OST0014_UUID 57.7T 38.0T 16.7T 70% /astro[OST:20] astrofs-OST0015_UUID 57.7T 37.1T 17.6T 68% /astro[OST:21] astrofs-OST0016_UUID 57.7T 37.1T 17.6T 68% /astro[OST:22] astrofs-OST0017_UUID 57.7T 38.0T 16.7T 70% /astro[OST:23] astrofs-OST0018_UUID 57.7T 37.4T 17.4T 69% /astro[OST:24] astrofs-OST0019_UUID 57.7T 37.7T 17.1T 69% /astro[OST:25] astrofs-OST001a_UUID 57.7T 38.3T 16.5T 70% /astro[OST:26] astrofs-OST001b_UUID 57.7T 37.4T 17.4T 69% /astro[OST:27] astrofs-OST001c_UUID 57.7T 37.5T 17.2T 69% /astro[OST:28] astrofs-OST001d_UUID 57.7T 37.1T 17.7T 68% /astro[OST:29] astrofs-OST001e_UUID 57.7T 37.8T 17.0T 70% /astro[OST:30] astrofs-OST001f_UUID 57.7T 37.8T 16.9T 70% /astro[OST:31] astrofs-OST0020_UUID 57.6T 34.6T 20.2T 64% /astro[OST:32] astrofs-OST0021_UUID 57.6T 34.3T 20.5T 63% /astro[OST:33] astrofs-OST0022_UUID 57.6T 34.9T 19.9T 64% /astro[OST:34] astrofs-OST0023_UUID 57.6T 33.6T 21.1T 62% /astro[OST:35] astrofs-OST0024_UUID 57.6T 33.5T 21.2T 62% /astro[OST:36] astrofs-OST0025_UUID 57.6T 35.0T 19.7T 64% /astro[OST:37] astrofs-OST0026_UUID 57.6T 33.7T 21.0T 62% /astro[OST:38] astrofs-OST0027_UUID 57.6T 34.1T 20.6T 63% /astro[OST:39] astrofs-OST0028_UUID 57.6T 33.5T 21.2T 62% /astro[OST:40] astrofs-OST0029_UUID 57.6T 33.6T 21.1T 62% /astro[OST:41] astrofs-OST002a_UUID 57.6T 34.2T 20.5T 63% /astro[OST:42] astrofs-OST002b_UUID 57.6T 33.8T 21.0T 62% /astro[OST:43] astrofs-OST002c_UUID 57.6T 34.9T 19.8T 64% /astro[OST:44] astrofs-OST002d_UUID 57.6T 34.1T 20.7T 63% /astro[OST:45] astrofs-OST002e_UUID 57.6T 33.8T 20.9T 62% /astro[OST:46] astrofs-OST002f_UUID 57.6T 34.5T 20.2T 64% /astro[OST:47] filesystem_summary: 2.7P 1.5P 1.0P 60% /astro
This gives a breakdown by OST and a summary at the bottom.