1st July 09:00:
We have discovered what we believe to be an imminent raid failure on this server, one of the disks has gone offline completely this morning and is not responding at all, while the other disk is up and showing no logged issues the write speed is incredibly low, <1 MB/s while the read speed is as expected.
As a result and due to the suddenness of this over 2 disks we cannot rule out hardware failure meaning we need to take immediate action.
The nightly backup did run as expected and we have a live image of all customer disks from 3am on a different local disk array so risk of absolute dataloss is mitigated.
We will now be placing the server in to maintenance mode and shutting down all running virtual servers while we start to migrate them to another physical server.
As they are migrated they will be brought back up however the entire process may take several hours.
Apologies for the inconvenience.
1st July 13:26:
All servers have been migrated to alternative nodes, if you are experiencing any issues please open a ticket.
Friday, July 1, 2022