12:30 - 16th Nov - The network issues in Phoenix are known, it has been escalated to the Network Ops guys and hopefully, it will be resolved soon. Sorry for the inconvenience.12:47 - 16th Nov - The issues were resolved a few minutes ago, the root cause was a locked up line card which impacted some of the VLAN's in the DC, this has been fixed ... Read More »
Due to an OpenVZ issue, there are currently over 30k unkillable processes running on the node which has impacted performance to the degree an emergency reboot is required, sorry for the inconvenience.
Services should be back up by 13:15 UK time, reboot Triggered 12:50 UK time.
Updates regarding the maintenance of node1 and 5 will be placed here. Update 1# 10:45am - Node 5 has been started early due to stability issues, due to the additional time it is expected to take to wait for this server to shut down gracefully that has been started early to avoid delays.Update 2# 12:25pm - The maintenance is complete all ... Read More »
There has been a server hang impacting some customers, the server is back up now and virtual servers are starting.
A crash dump was created and will be analyzed to find the root cause and any necessary work will be done for stability.
Sorry for the inconvenience.
The powerdns cluster has failed beyond recovery, it is being restored from backups now.
Any changes made in the last w hours may need to be re-added.
Sorry for the inconvenience it is expected to be resolved by the end of the day today.
UPDATE: The work has been complete.
12th June 2017 @ 15:30 - The issues on node 5 are known, they are being worked on, service should be restored soon.
12th June 2017 @ 15:51 - The containers needed to be stopped and the node was rebooted, service should be restored fully within minutes, further root cause investigations will continue.
09th June 2017 @ 11:38 (UK Time)The node is starting to hang due to issues with the Xen network stack, guest servers are bing shut down to clear this issue and while this is being patched the opportunity is being used to apply all outstanding XSA security patches.The server will be back up within 30 minutes, sorry for the inconvenience but ... Read More »
As many of you on this node are aware some performance issues have been present periodically.
The node hung around 15 minutes ago so the opportunity was taken to upgrade the server, unfortunately only half the ram arrived but it was put in to see if it helps.
Service will be restored within 15 minutes.
16:10 - 2nd May - The issues in the UK affecting some customers are known, it is being investigated everything should be back up within 30 minutes.
Sorry for the inconvenience.
10:20 23rd April: The issues with connectivity in the netherlands are known and are bing investigated.
10:35 23rd April: The cause was a customer being hit with a DDOS attack, this has been mitigated.