Update #7 09/17/2019 16:20 EST: all accounts except 2 large (over 30 GB) have been restored. Large websites will be restored manually.
We're currently finalizing restore: looking into each account individually to see if there any 500 errors associated with old scripts. Once again, if you do see 500 error, please:
1. make sure that account configuration files are pointed to /home/username. If you see /west/username, account may not work.
2. Default PHP version within cPanel: 7.2. We've installed old depreciated 5.6, please try it. It's only a temp fix, cPanel will drop it completely by the end of 2020. There are no security updates for it so please use it with caution.
Update #6 09/17/2019 14:25 EST: more than 50% of cPanel accounts were restored successfully. If you can't login with existing password, please try again later. If you see 500 error or PHP errors, please login to cPanel > MultiPHP then try to downgrade to PHP 5.6. Note: 5.6, 7.0 and 7.1 have been depreciated and shouldn't be used. Default secure version of PHP: 7.2.
Update #5 09/17/2019 11:52 EST account restore has been initiated.
Update #4 09/17/2019 8:46 EST. We've verified integrity on backup drive. There are no issues. All data is safe. We're currently moving fresh backups (09/15/2019 - 09-16/2019) to our offsite server just in case. After that we'll re-configure cPanel, CloudLinux and then install firewall, SSL etc. ETA: 2-4 hours. As soon as we complete cPanel setup, we'll roll onsite restore. We'll post ETA and progress here.
In the meanwhile, old server (panther), we'll stay online until 09/24. We'll monitor new server very closely in order to avoid any downtime in the future.
Thank you for your patience. We truly appreciate it.
Update #3 09/17/2019 7:51 EST hardware replacement has been completed (both disks have failed in RAID-1 array). We're currently loading new OS with cPanel. Backup drive is in good state (not affected by failure because it was out of RAID system) and we hope to recover all data.
If we can't recover it, we'll switch all cPanel accounts one by one back to "panther" because it has over 70% of all websites. After that we'll use offsite backups in order to restore other missing websites. Please be prepared: this process is slow and we'll need cPanel usernames for all missing accounts, starting with main one (with access to WHM). Thank you for cooperation.
Update #2 09/17/2019 3:35 EST: hardware check has been initiated. ETA: 10-16 hours. This software based check + manual checks will go through every piece of hardware: motherboard, processor, NVMe disks etc. If there something need to be replaced, it will be replaced by data center technician.
We understand that downtime is severe. We've noticed that some websites (around 300) are still on old working server (panther). It's possible to switch IP via DNS. Please contact us if you cannot wait until tomorrow. Unfortunately, we cannot switch every website to old server, only few. If issue related to failed hardware, for example, failed NVMe SSD, it will be replaced in 2-4 hours then we'll load backups once we get into the cPanel server.
Once again, thank you for your patience and understanding during this emergency maintenance. We really thought that brand new server with fastest NVMe disks would not fail in 72 hours.
Update #1 09/16/2019 23:13 EST: we're still trying to reboot server. It went online for about 2 minutes then got disconnected. We're currently checking every piece of hardware manually. It seems a hardware issue either with NVME disk or Intel processor. We'll post details soon.
One of our servers (LION) unexpectedly went down. We've already tried to reboot it but it got stuck. We're currently checking server hardware for any clues. There is no ETA yet but we'll post updates here as soon as we get more information.
Sorry for the inconvenience and thank you for your patience.