It's more than a simple outage, as VPS customers have seen their businesses erased through 123reg's mass deletion.
A case of fat fingers or dodgy code? We don't know -- but it's likely someone in 123reg's VPS hosting department will be looking for a new job soon.
Over the weekend, 123reg, known for providing cheap domain name registration and hosting, messed up. In what is being considered a mass deletion spree, an unspecified number of virtual private server (VPS) web hosting clients have realized their websites have faded from existence as if they never were to begin with.
For large corporations and SMBs alike, the eradication of a business' website without warning can be incredibly damaging and cause chaos for months to come -- a problem 123reg is going to have to face and explain to a horde of angry clients.
At the least, hundreds are in trouble. However, it could be thousands.
The company said in a status update on 16 April that an unspecified problem beyond "connectivity issues" and the "performance of our VPS product" was being investigated by engineers.
In a flurry of updates, 123reg said "all hands were on deck" to fix the problem, and the issue has been damaging enough to bring in external help.
123reg's Twitter account has been incredibly vague, which is likely to incense companies which are now hobbled -- or put on ice -- without a digital profile. The firm says that engineers are working on the problem, and VPS packages are being restored "using data recovery tools," which means that customer data has been wiped, somehow.
The advisory says:
"We are attempting to recover your packages bit by bit, while checking data integrity at the same time.
The process is quite time-consuming and the undergoing work is being done in parallel on multiple servers, in several locations.
Due to the nature of this process, we are unable to prioritise the recovery of specific hosts, but we are focused on restoring bulks of data at the moment. Progress is visible, and we have already managed restoring a certain number of hosts. However, we are unable to estimate when will the recovery be finished on all of the affected packages."
To make matters worse, 123reg is asking customers with backups to fix their own problem (although 123reg has been nice enough to provide a basic how-to guide).
One customer, software firm InnMaster, told The Register that on Saturday morning, someone from the web host "ran a script that had a catastrophic error in it," resulting in the deletion of servers and customer websites.
"Our hosting provider still hasn't restored anyone's data and most of their customers' websites are still offline," the company said in a statement. "They have told all customers that they would be better off starting from scratch with a blank server and rebuilding it, which is what we are now doing. Building a new web server from a fresh install is a full days' work."...
