email sent to all impacted users:
As you may have observed between midnight and 3am UK/London time the service on KVM10(node10) is becoming seriously degraded, it is being caused by the disaster recovery backup system.
Due to the obvious impact this is having for a fairly long time every night it has been stopped, while these were only ever intended as disaster recovery measures in the event os a raid or other hardware failure you should not consider to be running at a slightly elevated risk and it is important that you ensure your own backups are functioning as expected.
Some monitored trials will take place during the week on an alternate system as well as further investigations to find out why this specific node is responding differently to all others with the same backup system.
Sorry for the inconvenience, all efforts will be made to restore the service to full and remove any risk.
For clarity the node itself is still being backed up nightly, it is just customer disk images that will not be backed up until this is resolved.
23rd December 2020
10:45 am (UK/London) A test backup is running now using a new method, this will be allowed to complete as long as it is not more disruptive than the previous method so a fresher backup exists before the Christmas period begins.
Sunday, December 13, 2020