An uninterruptible process related to the LVM metadata has got stuck in a loop on the server which has bee silently eating up available ram and CPU time on the node, the issue causing it to loop has been resolved however it may take several hours for this to self right.

It will be assessed again in the morning, improvements can already be seen, if things are not as expected in the morning a full node reboot will be required and notice will be emailed to everyone.

Sorry for the degraded performance, sadly it looks like a few customer servers may have hung as a result of this, there should be no issues booting it again from the solusvm control panel however if it is possible to wait until after 09:00 UK time someone will be available to assist should there be any issues.

00:05am 30th January 2020
It is apparent that the issue is not getting any better, a full emergency reboot is being performed, all services should be restored within 30 minutes. 

00:16am 30th January 2020
The reboot is complete and servers are starting.

Wednesday, January 29, 2020

« Back