Good Morning,
The issues impacting Stor3 are known, all access was lost a little while ago, the remote KVM/IPMI device is also unresponsive, remote hands are currently investigating. When more updates are available they will be placed here.
If the server requires a power cycle and this has been a software-related event then it is likely that all virtual servers will then run an fsck, this being the case performance will be degraded for some time when it returns as this is never quick on such massive storage arrays.
Updates to follow, sorry for the inconvenience this has caused.
UPDATE: 09:04 (UK/London) the issue has been identified, one of the fans has physically failed in the server which has resulted in overheating, it was possible to bring the server up for a very short time to confirm the OS is intact which on the very quick investigation it seems to be, there are replacement parts on route now.
It is highly likely an fsck will be required however, more updates will follow once the physical maintenance is complete.
UPDATE: 10:39 (UK/London) The hardware swap has been completed, the server is booting now, it looks on the surface that no data was impacted although due to the sudden power down some local filesystem issues may be present in the guest KVM servers, it seems many of them have picked this up and are fsck'ing based on console outputs however the vast majority have simply started up as expected.
Due to the fsck's running, performance will be degraded for a while but will return to normal.
It is recommended that all customers set their file systems to do a consistency check and reboot to correct any underlying issues this may have caused but that is entirely optional.
Friday, November 27, 2020
