Initial indications indicate a hardware failure so it may be a disisaster recovery scenario in which case all efforts will be made to restore services within 24 hours.
Updates will follow as they become relevant.
Update 20:27 - Data has been recovered which means it is not necessary to revert to the backups taken at midnight, restoration is in progress and is hoped to complete by midnight (UK/London)
Update 21:15 - It is expected that Node 12 services will be restored around 22:30 (UK/London)
Update 22:25 - The previous estimation for restoring Node 12 services has proven hard to deliver, this has been pushed back to midnight for Node 12 and 2am for KVM3
Update 01:08 - The services have been restored - a full anonysis will take place over the comming week and an RFO sent to those impacted.
Update 09:10 - Due to issues with the solusvm database during the recovery efforts the OS can no longer see volumegroups on the server cauing it to become unstable, a reboot has been performed as a last resort.
Update 09:22 - There were contant issues from the recovery of Node 12 as such the only option has been to start the disaster recovery from the last backup, this is in progress you may lose around 12 hours of activity, the restore is expected to complete by 13:00 today (UK/London) and that will be the end of the issues, KVM3 is now considered stable.
RESOLVED 13:30 - The services were fully restored by 13:30, no further disruption is expected, and RFO will follow as soon as it is possible.
Tuesday, January 22, 2019