There has been a raid failure on kvm4 in London, while the server is still accessible but barely functional we are attempting to recover customer disk images to restore on another node.
We do not have time scales now but will attempt to restore service as soon as possible.
In some cases, you may be required to restore from your own backups.
Sorry for the inconvenience.
Update 11:32 UK/London time - 17th December
Some disk images are corrupt, we are recovering what we can with ddrescue, so at least some data may be available if you boot to recovery mode, in some cases that will sadly be 0.
We are now prioritizing restoring access and service so people can start to recover from their backups.
We will investigate this further once the service has been restored, we do not want to reboot the server until we have recovered as much as we can as it may never come online in any usable way again.
Thank you for your understanding.
Update 12:11 UK/London time - 17th December
All data that could be recovered has been recovered, and the majority of servers are back online however a handful were unrecoverable, please reinstall and recover from your backups.
Please also note that there is a known issue with VNC right now, please ensure your passwords are set to 8 characters a-z A-Z 0 -9
Update 12:34 UK/London time - 17th December
We did manage to get the array online again, we are attempting a fresh recovery just in case we can recover the few that previously failed
Update 13:08 UK/London time - 17th December
We believe we have recovered the remaining virtual servers, a few images held no data but were error-free so they may have been idle servers anyway with no data there is no way for us to know.
The last images are being copied to the replacement server now, which should be complete within 20 minutes, we are marking this as resolved.
Sunday, December 17, 2023
