Good Morning,
The issues on Netherlands node 3 are known and are being worked on, yesterday a drive was replaced in the raid array and today another disk has dropped off before the full sync had completed.
Things are being assessed now to see if this is recoverable.
Updates will follow here.
Update 1# Recovery is being attempted, this may take a few hours to complete, if this fails there will be no choice but to replace the raid array completely and recreate everyone's server so you can restore from your own backups.
Update 2# The recovery has around 140 minutes left, an fsck will be required on the host node os partition and there will be a better understanding of where things are at that stage.
Update 3# 19:18 GMT+0 - The recovery process is almost complete, around 10 minutes left to go and then it will be clear if it is worth spending any more time on or simply re-creating everyones VPS fresh and allowing you to restore from your own backups.
Update 4# 19:36 GMT+0 - Data seems accessible so full data recovery is worth the time, the LVM data is copying over to the fresh drives, the raid will be synced and boot will be attempted.
Expected time for first boot: 22:30 GMT+0
Sadly hardware failures are unavoidable this node had an uptime of 873 days prior to any hardware specific issues taking it down completely and has already had 3 disks replaced over that time.
Update 5# 23:25 GMT+0 The node is now booted, all customer data has been recovered it is impossible to rule out some loss however that does not seem to be the case, sadly in the final 2% of the raid rebuild one of the drives produced a fault so for safety it has been removed and will be replaced when engineers are available in the morning.
Update 6# (20th Feb) sadly the instability has returned, all virtual machines are being shut down in order to stabilise the node, sorry for the inconvenience, work is expected to last 2 - 3 hours.
Update 7# 20th Feb @ 22:34 GMT - The server has been recovered, due to the recent issues with this server and the age of some of the others the decision has been made to purchase new equipment for long term resolution and stability, a lot of the servers in the Netherlands are beyond or approaching 3 years continued service so the refresh is about due anyway.
A general customer email will be sent out about this when an ETA is available and full plans can be made.
Sunday, February 15, 2015
