29th Nove 2020 - 16:37: Due to a number of hung unkillable containers on the node, a reboot is required, service will be resumed as soon as possible.
Sorry for the inconvenience.
The control panel (SolusVM) that manages the nodes has managed to generate over 30,000 processes stuck in wait, all taking up CPU time. this has been reported upstream as a bug. A fix has been launched however killing 30,000 LVM based processes has put the server under significant further load, it will be left for 30 minutes to complete however ... Read More »
The connectivity issues on node10(KVM10) are known and being investigated.
Updates to follow.
Good Morning, The issues impacting Stor3 are known, all access was lost a little while ago, the remote KVM/IPMI device is also unresponsive, remote hands are currently investigating. When more updates are available they will be placed here. If the server requires a power cycle and this has been a software-related event then it is likely ... Read More »
08:42PM - 05th September Due to significant performance issues that have been traced back to a hung container that is causing significant disk I/O issues node VZ27 needs an immediate reboot.This will take place as soon as this announcement goes live and hopefully, everything will be back up within an hour.Those that were on OpenVZ 6 that got ... Read More »
The issues are known, things may take a while to respond or actions may say they are done when nothing has changed e.g. reinstalls reboots, VNC is also impacted.
This is an intermittent network issue outside of our network boundary, communication has been sent to the responsible party and hopefully, this will be resolved within 24 hours.
Below is the email sent to all impacted users, updates and progress when work begins will be below the email:Good Afternoon, Recently new configuration of storage nodes was deployed after learning lessons from the older hardware configurations and limitations. The difference in performance between the old and new configs is quite literally night ... Read More »
Issues impacting the VZ27 UK server should be resolved within 45 minutes, the service is restarting now.
It is possible you may need to manually boot your server if it does not come back up on its own by 10:00 (UK/London)
Sorry for the trouble this was a silent failure so it was not picked up until the morning checks.
The network issues in Phoenix are known, the local DC network team are investigating.
16th July 2020: 08:30 (UK/London) There seems to be a hardware failure on this node, the network card is not being detected at all by the OS.Remote hands have been requested and acknowledged.ETA for recovery at this stage is unknown.Sorry for the inconvenience, updates will follow as they are available. 16th July 2020: 10:10 (UK/London)The issue ... Read More »