Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Warning
titleFor transfering large amounts of data

NEVER USE THE LOGIN NODES for transferring large amounts of data. Users should use the datamover nodes for this purpose. These nodes are accessible through the generic hostname hpc-data.pawsey.org.au or through the "copyq" queue for batch processing through scheduler.

As these nodes are separate from the login nodes, the load for transferring large files or a large amount of data will not impact other users.  All standard filesystems, such as /group, /scratch, /astro and /home, are accessible from these data-mover nodes.

A second warning: if the "group" property of transferred files/directories is not set as your project

...


...

Warning
titleIf the "group" property has been overriden

By default, your personal directories in the file systems /scratch and /group (easily accessible with the environmental variables $MYSCRATCH and $MYGROUP) are set to belong to: "yourUsername-yourProject" in the "owner-group" properties. This combination of ownership properties has been defined through set-gid. Also by default, all the files and subdirectories created under your personal directories in /group and /scratch will inherit the same ownership properties: i.e. should belong to: "yourUsername-yourProject".

Unfortunately, some file transfer programs override our set-gid defaults for ownership properties. The typical unfortunate change is to set "yourUsername-yourUsername" instead of "yourUsername-yourProject" for the "owner-group" properties (see example below). This makes the files abide by the quota restrictions as set on /home and won't be able to make use of extended quotas as on /group, /scratch or /astro. You may receive errors during an upload if this is the case.

...