22nd May 2023

We are aware of the outage of some services in Phoenix, this seems to be due to a raid failure on initial investigation.

We are attempting to recover this.

Updates to follow

---

09:45 22nd May 2023

Unfortunately, the raid card is unable to recover the configuration, the cachecade device is switching from offline to online and additionally, disks are just vanishing from the array between reboots.

We have requested remote hands to physically attend the server to resecure all devices and replace the cachecade altogether, it is possible we can recover the array from here, but the chances are not great, customers are advised to start preparing their disaster recovery plans.

In the event that an engineer on site is unable to recover the hardware will be replaced and fresh services will be provided to customers to recover from your own backups.

We feel at this stag it is worth giving the engineers a few hours to try and recover this rather than immediately implementing a full replacement.

Updates to follow as they become available.

 

---

 

11:50 22nd May 2023

Despite the best efforts the engineer on site was unable to recover the array, the card has been replaced and we will proceed to provide new virtual servers for customers.

An email will be sent to those impacted soon.

---

Copy of email sent to all impacted customers:

Good Morning,

 

I hope you are well.

Further to the announcement here: https://clients.inceptionhosting.com/index.php?rp=/announcements/288/Phoenix---Raid-Failure---22nd-May-2023.html we have not been successful in our efforts to recover the array.

As such, we will replace the hardware and provide new servers for customers to self-restore from their backups.

In such cases when completely reprovisioning fresh servers, we cannot guarantee the same IP address will be given; we will also be, where possible, provisioning servers on the Clouvider network instead, as that gives us significantly more control in such cases as well as the hardware and network being a significant upgrade.

This has happened on some of the last remaining legacy hardware used by Inception Hosting within IOFlood. Hence, it makes sense to use the opportunity to move people where possible to the new network owned by the parent company Clouvider to add DDOS protection, native IPv6 and 10 Gbit network ports. 

We will now focus on restoring services, so some support ticket replies may be delayed while this takes priority.

Thank you for your understanding, and apologies for the inconvenience. 

Anthony.
Inception Hosting Limited.

 

---

Final Update

Services have been restored and customers have been sent details.

If anything is not as expected, please open a ticket.

As a reminder, as we have already had a number of tickets, we do not backup customer data; your data is your own, this has always been the case, and no one has been paying for managed backup services; if your question is regarding data recovery, please contact your system administrator to ask them where your offsite backup is located.



Monday, May 22, 2023

<< Geri