Page tree
Skip to end of metadata
Go to start of metadata

Status:

RESOLVED

Points of Contact:

 help@pawsey.org.au 

Start Date/Time (AWST):

07:00

Responsible:

Chris Schlipalius (Data Team Lead)

Estimated End Date/Time (AWST):

17:00

Accountable:

 Mark Gray (Head of Platforms) 

End Date/Time (AWST):

11:00

Informed:

Summary:

Banksia S3 and https services unavailable (RESTAPI).

Systems/Services Affected:

Banksia (Pawsey Offline/Cool Data Storage), MWA Archive/ASVO, Pawsey Data Portal/Mediaflux.

Updates:

  • After maintenance and after client-side services were resumed, issues with our Banksia certificate were detected this morning.
  • New certificates were soft deployed to the servers but this seems to have disagreed with Banksia services.
  • A critical priority ticket has been lodged with the vendor.
  • We have investigated and are about to test a remedy for the issue.
  • The issue has been identified and a fix deployed to just vss-6 for now. Initial testing verifies the fix for certificates is working. After further verification via a client side test, this will be deployed to all Banksia nodes then a rolling restart of scoutam service and the s3 gateways will proceed.
  • Client verifies all is working with vss-6 (S3) and we have deployed the fix to all nodes and performed a rolling restart of services.
  • The incident is now resolved. Service account clients have been contacted to ensure they renew their S3 session connections as well as their REST API tokens (as the separate service providing these has had it's certificate renewed).

Post-Incident Summary: