[UPDATES AT THE BOTTOM]
Good Afternoon,
As some of you know and will have experienced there has been a fair bit of disruption in the Netherlands recently on the Xen platform, most of which has been hardware related along with some network related issues, the current hardware in most cases has been in service over 3 years and in recent months there has been a lot of disk failures, Motherboard failures, KVMoIP failures and a host of other problems, all of the Netherland nodes are also on CentOS 5 and do not benefit from the enhancements of 6/7 in terms if Disk IO and fully functional ebtables etc to prevent IP stealing/spoofing from malicious customers.
To further complicate things there has been increased delays in hardware swaps, there are now only day time on-call staff available to go to site through the provider Inception Hosting Limited uses and there is no longer an SLA worth having from the providers used in the data centre which is not a favourable position to be in, this decision was made after it recently took 7 days to replace a disk as there was firstly no staff available to swap it and when it was replaced finally it was replaced with a drive that had been in use for 3 years previously and had bad sectors which almost resulted in complete data loss and this was not an isolated incident.
As such the decision has been mode to move to our own hardware and IP space in the Netherlands to ensure long term stability of service, the physical location of your service will not change however the overall experience should come on leaps and bounds once the migration is complete, the hardware will be with i3d.net directly instead of a sub provider with leased racks, with staff on site 24x7 and a real SLA that can be relied upon and faster uplinks per server.
The new host nodes will be on the latest kernels and Xen versions for increased performance and stability, Xen HVM will be added in the coming months and all servers will be 12 disk Raid 10 arrays for greater redundancy and performance.
The downside from a customer perspective is that an IP change is unavoidable, and inbound bandwidth will now be metered, over the next 48 hours you will see a new IP in the 95.46.x.x range in SolusVM, please take some time if possible to configure your new IP in advance along side your current one so you do not end up losing connectivity, you will also be getting an IPv6 subnet rather than individual addresses, if you do not use IPv6 at all then this is of no concern.
Servers will be migrated in small batches to minimise disruption, migrations will start on the 10th March the priority will be given to the less stable nodes currently Nodes 3, 4 and 6.
Please take some time to reboot your server before the migration, experience has shown that many people have not rebooted in a very long time and due to human error they may not boot up properly and this can cause issues and delays with the migration while trying to find the root cause with a higher number of variables that usual.
Packages and prices will remain the same with the exception of a handful the very old ‘Grandfathered’ packages which will be matched to the closest available package (no price changes), an email was sent out around 3 years ago regarding this however it is easy to tell if this affects you as your package will have a name that is not in the formal of either EU256 or NL1024 i.e. location followed by the amount of Ram.
As always your patience and understanding is appreciated, while many of the Netherlands Xen nodes are approaching 3+ years of uptime the hardware is old and restrictions with the current host within the data centre are now prohibitive in terms of a continuing long term stable service.
I appreciate this won’t be a welcome email for everyone but as always I am happy to respond directly to any concerns.
Thanks.
Anthony Smith.
Inception Hosting Limited.
Update 1# 10th March @ 20:30 - Node 3 migrations in progress.
Update 2# 12th March @ 01:50 - Migration on all nodes is around 60% complete, work will continue tomorrow, there are some rDNS issues which should be reolved within 24 hours, sorry for the inconvenience.
Update 3# 13th March 03:00 - A bug has been found relating to solusvm and the latest xen version, due to the forced XL toolstack for Xen it has been necissary to restart all servers on the new Node 1 server, service will be restored soon.
I understand this is not ideal however it was required to maintain performance and stability and was better to do now at near the end of all the migrations than later when it started to cause issues.
Update 4# 13th March 04:21 - sadly the whole server needed to be rebooted, which has done, servers are staring in order.
RDNS will be available in the next 12 hours.
Update 5# 13th March 14:30 - there are some serious compatibility issues that could not have been forseen with the latest Xen versions, sadly every attempted fix requires a node reboot, this is being dealt with and tickets cannot be answered right now as this is the priority, please be assured that this is being worked on however there is no quick fix.
Update 6# 13th March 18:27 - all servers have booted now and have been running for around an hour, monitoring will continue.
Update 7# 13th March 20:30 - everything has remianed stable, rDNS for ipv6 is now functional, IPv4 should follow very soon.
Update 8# 13th March 23:00 - A kernel oops was triggered, luckily the logs have been on watch for the last 3 hours so the exact event that caused the issue was spotted as it happened, the bad news is that the entire server needs to be rebooted again as the xen networking drivers have crashed as a result.
Update 9# 14th March 00:22 - Most VPS has boot now the remining ones should be up within 30 minutes.
Update 10# 14th March 02:35 - Further issues have been found, the server requires another reboot, sorry for the inconvenience.
Update 11# 16th March 14:30 - The issues on NL Node 1 have been resolved and things have remained stable for 48 hours+ there will be 1 final reboot on Saturday 21st @ 00:05 to replace the battery on the raid card which is not fully charging and that should be it.
rDNS is now available for IPv4 please click on the network tab in solusvm which will allow you to set your own rDNS (forwrd zone must be present)
Update 12# 20th March 22:10 - Due to a miss communication the work due for 00:05 has begun early, sorry for the inconvenience.
Update 13# 21th March 22:25 - Raid battery replaced, host server and virtual machines booting now.
Sexta-feira, Março 6, 2015
Powered by WHMCompleteSolution
