An email regarding security patching was sent out today, the reboot has been done however an fsck has been forced so it may take a while longer than expected to get things back up and running, sorry for the unexpected delays.
Update# 18th January 23:32 GMT - what should have been a routine patch and reboot has turned in to a disaster, the fsck failed to complete, it seems there may have been a silent failure within the storage, the exact cause is unknown, smart tests have been ran on all disks and they pass without error. the LVM is currently being transferred to ensure your data is safe however with these sorts of volumes of data there is simply no way to achieve this fast, current estimate based on capturing 5 minutes of progress puts the estimate of 4 hours and 47 minutes to complete.
Further investigation cannot be done until this is complete there is no way to speed it up, I will monitor the progress and update accordingly.
Update# 19th January 02:40 GMT - the transfer is at 75%
Update# 19th January 04:19 GMT - The restore is taking place now, hopefully service will be resumed within a few hours.
Update# 19th January 09:05 GMT - Full recovery has been completed and virtual servers are booting now.
Wednesday, January 18, 2017
