Page tree
Skip to end of metadata
Go to start of metadata

Status

RESOLVED

Start Date/Time (AWST)

 

Estimated End Date/Time (AWST)

 

End Date/Time (AWST)

 @ 12:45

Summary:Read Only "/askapbuffer" on "askapfs1-OST0027"
Systems/Services AffectedAny system using /askapbuffer on storage volume "askapfs1-OST0027"


Points of Contact

  • help@pawsey.org.au


Responsible:

  • Ashley Chew


Accountable

  • Ashley Chew


Informed:


Updates:

  • Pawsey researcher experiencing read or write issues to "/askapbuffer"
    • Issue GS-14044 for reference
  • Storage Volume "askapfs1-OST0027" would become read-only ie
askapfs1-OST0026_UUID 61925523668 46256574476 12546493940  79% /askapbuffer[OST:38]
askapfs1-OST0027_UUID 61925523668 43696418548 15106642048  74% /askapbuffer[OST:39] R
askapfs1-OST0029_UUID 61925523668 46749991988 12052993212  80% /askapbuffer[OST:41]


  • HA Failover would restore the storage volume as being read / write
  • Tests confirmed there the issue would arise if you wrote to certain sections of pre-allocated storage by lustre from "askapfs1-OST0027" where it would fail and subsequently put the volume in Read Only mode
  • Errors confirmed on filesystem ie

    askapfs1-OST0027: Directory inode 30146561, block #5, offset 0: directory corrupted
    askapfs1-OST0027: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
  • e2fsck was run on the storage volumes to remove and fix inconsistencies on filesystem
  • "askapfs1-ost0027" was restored along with the storage volumes pair in HA configuration


Post-Incident Summary: