We are aware and investigating the cause.
This is confirmed as a network issue that seems to be affecting inbound traffic on tinet, the NOC have been alerted.
UPDATE 1: all connectivity has been restored however during the investigation and restoration the PDU for one of the racks was tripped while moving the switch as a result poeer has been lost to Node 8, the onsite team are working to restore everything once power has been restored further investigation can be done to attempt the recovery of Node 8, the server runs Raid 10 so data loss is not expected.
UPDATE 2: Power has been restored to Node 8, the VPS's are booting up now however the raid is recovering at the same time, no data loss has occured to the host node OS and everything seems to be starting up, loads and disk wait may be high for an hour or so while this goes on so your patience is appreciated.
UPDATE 3: It seems there are some issues with Xen on the node, investigations are underway, the server may require a reboot, please stand by for further updates.
UPDATE 4: The Xen packages may need to be reinstalled, please expect a number of reboots.
UPDATE 5: Xen has been removed and reinstalled, the virtual servers are now booting, the load is incredibly high right now on the node while all servers probably run fsck things should setting withinin an hour or so, please avoid doing any OS reinstallations at all costs right now.
Friday, June 7, 2013
