Page tree
Skip to end of metadata
Go to start of metadata
On this Page

Introduction


The Astonomy Filesystem /astro is a lustre filesystem provided for the scratch space needs of the MWA group who perform computations on the Garrawarla cluster. It is an SGI/HPE provided cluster of nodes backed by DDN storage. The system currently contains 2 Metadata servers (MDS) with 2 Metadata targets (MDT). It has 4 Object Store servers (OSS) for storing data and they have 48 Object Store targets (OST). This gives approximately 2.7 PB of usable storage. It has a possible read and write speed of over 10GB/s and Pawsey staff have been easily getting 7-8GB/s when using only the four copyq nodes to transfer data around using dcp. The single threaded IO speed is greater than on /scratch2 due to the updated version of Lustre. 

The expandability of lustre means that the filesystem can be expanded, without downtime, by adding more OSS's and OST Disk behind them in groups of 2 (for high availability).

Location


The Astronomy filesystem is mounted on all Garrawarla, Magnus, Zeus, Topaz and Galaxy nodes and data mover nodes as /astro. The top level directory has directories for all the areas that /astro has:

garrawarla-1:/astro # ls -l /astro/
total 20
drwxrws--- 36 root mwaeor     4096 Oct 28 09:52 mwaeor
drwxrws--- 12 root mwaops     4096 Sep 30 16:31 mwaops
drwxrws--- 47 root mwasci     4096 Dec 11 16:56 mwasci
drwxrwsr-x 57 root mwavcs     4096 Dec 16 16:46 mwavcs
drwxrws--- 27 root pawsey0001 4096 Dec 17 12:22 pawsey0001

The pawsey0001 directory is for Pawsey testing of the system and can be set up in different ways as needed, it will not often be used.

Quotas


At the time of writing MWA have requested that mwaeor, mwavcs, mwaops and mwasci are assigned 370TB, 580TB, 20TB and 600T respectively.

To check the current quota use:

lfs quota -g projectcode /astro

Usage


To check usage you can use the normal unix df command to check the entire filesystem. 

galaxy-1:/astro # lfs df -h /astro/
UUID                       bytes        Used   Available Use% Mounted on
astrofs-MDT0000_UUID      542.1G       26.0G      479.4G   6% /astro[MDT:0]
astrofs-MDT0001_UUID      542.1G       32.7G      472.7G   7% /astro[MDT:1]
astrofs-OST0000_UUID       57.7T       26.6T       28.1T  49% /astro[OST:0]
astrofs-OST0001_UUID       57.7T       26.4T       28.4T  49% /astro[OST:1]
astrofs-OST0002_UUID       57.7T       25.9T       28.8T  48% /astro[OST:2]
astrofs-OST0003_UUID       57.7T       28.6T       26.2T  53% /astro[OST:3]
astrofs-OST0004_UUID       57.7T       26.9T       27.9T  50% /astro[OST:4]
astrofs-OST0005_UUID       57.7T       26.6T       28.2T  49% /astro[OST:5]
astrofs-OST0006_UUID       57.7T       26.3T       28.4T  49% /astro[OST:6]
astrofs-OST0007_UUID       57.7T       26.8T       28.0T  49% /astro[OST:7]
astrofs-OST0008_UUID       57.7T       26.8T       27.9T  50% /astro[OST:8]
astrofs-OST0009_UUID       57.7T       26.3T       28.5T  48% /astro[OST:9]
astrofs-OST000a_UUID       57.7T       26.6T       28.2T  49% /astro[OST:10]
astrofs-OST000b_UUID       57.7T       26.8T       28.0T  49% /astro[OST:11]
astrofs-OST000c_UUID       57.7T       25.6T       29.2T  47% /astro[OST:12]
astrofs-OST000d_UUID       57.7T       27.1T       27.6T  50% /astro[OST:13]
astrofs-OST000e_UUID       57.7T       27.0T       27.8T  50% /astro[OST:14]
astrofs-OST000f_UUID       57.7T       26.5T       28.3T  49% /astro[OST:15]
astrofs-OST0010_UUID       57.7T       37.3T       17.5T  69% /astro[OST:16]
astrofs-OST0011_UUID       57.7T       38.1T       16.7T  70% /astro[OST:17]
astrofs-OST0012_UUID       57.7T       37.5T       17.3T  69% /astro[OST:18]
astrofs-OST0013_UUID       57.7T       38.0T       16.7T  70% /astro[OST:19]
astrofs-OST0014_UUID       57.7T       38.0T       16.7T  70% /astro[OST:20]
astrofs-OST0015_UUID       57.7T       37.1T       17.6T  68% /astro[OST:21]
astrofs-OST0016_UUID       57.7T       37.1T       17.6T  68% /astro[OST:22]
astrofs-OST0017_UUID       57.7T       38.0T       16.7T  70% /astro[OST:23]
astrofs-OST0018_UUID       57.7T       37.4T       17.4T  69% /astro[OST:24]
astrofs-OST0019_UUID       57.7T       37.7T       17.1T  69% /astro[OST:25]
astrofs-OST001a_UUID       57.7T       38.3T       16.5T  70% /astro[OST:26]
astrofs-OST001b_UUID       57.7T       37.4T       17.4T  69% /astro[OST:27]
astrofs-OST001c_UUID       57.7T       37.5T       17.2T  69% /astro[OST:28]
astrofs-OST001d_UUID       57.7T       37.1T       17.7T  68% /astro[OST:29]
astrofs-OST001e_UUID       57.7T       37.8T       17.0T  70% /astro[OST:30]
astrofs-OST001f_UUID       57.7T       37.8T       16.9T  70% /astro[OST:31]
astrofs-OST0020_UUID       57.6T       34.6T       20.2T  64% /astro[OST:32]
astrofs-OST0021_UUID       57.6T       34.3T       20.5T  63% /astro[OST:33]
astrofs-OST0022_UUID       57.6T       34.9T       19.9T  64% /astro[OST:34]
astrofs-OST0023_UUID       57.6T       33.6T       21.1T  62% /astro[OST:35]
astrofs-OST0024_UUID       57.6T       33.5T       21.2T  62% /astro[OST:36]
astrofs-OST0025_UUID       57.6T       35.0T       19.7T  64% /astro[OST:37]
astrofs-OST0026_UUID       57.6T       33.7T       21.0T  62% /astro[OST:38]
astrofs-OST0027_UUID       57.6T       34.1T       20.6T  63% /astro[OST:39]
astrofs-OST0028_UUID       57.6T       33.5T       21.2T  62% /astro[OST:40]
astrofs-OST0029_UUID       57.6T       33.6T       21.1T  62% /astro[OST:41]
astrofs-OST002a_UUID       57.6T       34.2T       20.5T  63% /astro[OST:42]
astrofs-OST002b_UUID       57.6T       33.8T       21.0T  62% /astro[OST:43]
astrofs-OST002c_UUID       57.6T       34.9T       19.8T  64% /astro[OST:44]
astrofs-OST002d_UUID       57.6T       34.1T       20.7T  63% /astro[OST:45]
astrofs-OST002e_UUID       57.6T       33.8T       20.9T  62% /astro[OST:46]
astrofs-OST002f_UUID       57.6T       34.5T       20.2T  64% /astro[OST:47]

filesystem_summary:         2.7P        1.5P        1.0P  60% /astro

This gives a breakdown by OST and a summary at the bottom.


  • No labels