Update #4 November 29th, 2019 22:01 EST: all websites (467) have been restored on new server: MOUSE. Thank YOU for your patience!
Known issues (will be fixed this weekend):
- Softaculous APP installer is missing within cPanel;
- Large accounts (over 10 GB) might be missing within WHM > List accounts. Those accounts will be restored manually, one by one. It may take up to 24 hours to restore such data.
- Website may produce 500 "Internal server error". New server uses current MySQL and PHP. Your scripts might not be compatible with 7.2. In order to fix this issue:
1. Please login to cPanel then navigate "MultiPHP Manager".
2. On the right-hand corner please select PHP version: 5.6 then select account below and then hit "apply" button.
3. Wait for about 3 minutes (changes are not instant, it requires Apache reboot).
If you still receive 500 error, you may try different PHP version (for example, 5.5 or 7.0).
Note: PHP 5.5, 5.6, 7.0 and 7.1 should NOT be used for security purposes. Please keep your scripts upgraded! Current version is: 7.2. Newest available version is 7.3 (should be used for test purposes only). We do not recommend 7.3.
Update #3 November 29th, 2019 18:18 EST: Account restore will commence on new Dell server at 20:00 EST. Server ID will change from "LION" to "MOUSE". ETA: 4 hours. We'll update this article as soon as we complete account restore.
DNS will not change. Server (host) IP, if you use external DNS, must be replaced. We'll post all information soon.
Thank you again for your patience.
Update #2 November 29th, 2019 2:36 EST: "LION" server has been repaired. We've replaced both NVMe disks. OS has been reloaded, new cPanel/WHM installed. Your websites will be up and running by today evening.
At the same time, we are awaiting for new Dell server with different hardware configuration. All accounts will be moved to new, redundant server once we complete setup. This is necessary to resolve all issues for good.
We do appreciate your patience during this emergency event. We apologize for the inconvenience. We'll do our best to keep your new server online all the time.
Update #1 November 28th, 2019 6:53 EST: "LION" server didn't respond to automatic and manual reboot requests. We're currently running FSCK. New, replacement server has been ordered ahead of time because we had experienced similar event on other server with same NVME disks (same server setup in RAID-1). We want to avoid duplicate restore because it takes longer. We'll post all details soon.
Thank you for your patience! There is no need to call or submit a ticket with downtime report.
Do you have site backups?
Yes, we do.
When will you bring up my website?
There are two options: 1. fix current server (replace NVMe with regular SSD then OS reload and then estore) or 2. setup a new server with better hardware, load OS and then restore all websites. We'd rather wait for new server to avoid duplicate restore/transfers. Unfortunately, it may take longer than usual due to Thanksgiving. We'll do our best to prevent extensive downtime.
What happens with email?
If you have Google Apps or similar backup MX, there is no impact because we use redundant DNS. If you host email on the same server (lion), you need to wait until we restore all services, including cPanel and Webmail. During restore period all email may or may not bounce to sender (it depends on sender server configuration).
Do you have specific ETA (time to resolve the issue)?
Emergency window set for November 28th, 2019 6:00 EST - November 30th, 2019 14:00 EST. It may take longer to deliver replacement server due to Thanksgiving Holiday. That's why we're currently trying to fix current server.
I've experienced similar downtime event in September. I can't afford another downtime. Do you guarantee that new server will fix all issues for good?
Yes, because new Dell server will use SSD (data center edition) with different RAID setup, DDR4 and new dual Intel Processors (they won't overload during offsite backups). These servers are extremely reliable. We'll post details once we receive replacement server.
Thank you for your patience.
We're aware that "LION" server went down. Unfortunately, reboot didn't work out. We're currently investigating this outage. There is no ETA yet but we'll post more information soon.
Thank you for your patience.