Page tree
Skip to end of metadata
Go to start of metadata

Status

RESOLVED

Start Date/Time (AWST)

11:00

Estimated End Date/Time (AWST)


End Date/Time (AWST)

07:30

Summary:Instances cannot be access or created
Systems/Services AffectedNimbus


Points of Contact

  • help@pawsey.org.au


Responsible:

  • Cloud admins (Luca, Jeremy, Greg)


Accountable

  • Mark Gray


Informed:


Updates:

  • Mariadb database connection issues, high RAM usage
  • RabbitMQ connection errors
  • OpenVSwitch services not running properly
  • Neutron package minor version mismatch
    • Upgrades applied, services restarted
  • Instances being spawned on new unready hypervisors, despite host aggregate configuration
    • new hypervisors shut down
  • Issue seems resolved. Update provided on status page, status now Monitoring.


Post-Incident Summary:

Nova and Neutron packages, services and machines became incorrectly configured.

We applied updates and restarted services so that packages and services were correct, and shut down the machines which should not have been in the cluster.