Unfortunately a quick decision had to be made to reboot as it was clear the server was about to hang, this issue was known on this kernel version and has been seen before, sadly during the reboot it actually did panic so the VM’s were not cleanly shut down.
Everything is booting back up now and all VM’s should have experienced less than a few minutes of down time assuming your services were set to start up automatically in most cases you won’t have even noticed.
Again, sorry for the inconvenience.
Update #1 the server again hung immediatley after reboot, emergeancy maintenance work is being done to track down the root cause and resolve the issue.
Update #2 The server has been updated so no remains of the package versions causing the issue remain, VM's are booting and I am hopeful this has resolved the issue, due to a few unclean reboots the raid is syncing so disk IO may be slugish for a short period of time.
Anthony.
Friday, December 20, 2013
