Data Center Update

[Manny]

Effected clients, co-workers, partners:

At approximately 8 A.M. EST a segment of our data center experienced a power outage. Shortly after that we realized the city had experienced a power outage as well which attributed to the loss we saw in the NOC. We immediately went into generator power and we are still currently running on generator.

What happens when there is power loss is that we immediately go into UPS for a few minutes meanwhile the generator powers up. The UPS ensures no service disruption meanwhile the generator turns on. This happened normally but approximately 40% of the servers tripped during this process. Power loss and going into UPS followed by generator power has happened a few times in the past. But we have never experienced what has happened today - the power surge during this outage was extremely high.

We are meeting with our electrical engineers as we speak to analyze the situation to learn exactly what happened. We still have the data center running on generator as the power in the city could still be unstable and we do not want to risk any further outages. As many of you know these kind of issues do not happen often and we are taking this incident very seriously. The data center is equipped with a high level of UPS and generators are in place to ensure maximum uptime. We apologize for the inconveniences as we know this has effected a lot of you and your clients. We will be compensating per our SLA and will ensure we find out in detail what happened, resolve it completely and put measures in place so it doesn't repeat again.

All effected servers should be operating normally now and we sincerely thank you for your patience.

[Kansas Viking]

Thanks Manny. That explains everything this morning. Appreciate the heads up and good luck.

[rakel]

Thank you for the update. I've been really frustrated today, but I know I can't be as frustrated as you are!

[rustyz82]

While I am very thankful for the update, all affected servers are not operating normally. My dedicated server is still down and has been all morning.

[benjy]

All effected servers should be operating normally now and we sincerely thank you for your patience.

I'm a reseller on dime70, and I can't get to any of my or my clients' sites.

-benjy

[amycelona]

You are guys are normally fabulous... I knew it had to be something really big for you to go totally offline. I was scared (mostly it is being offline and not knowing why). Thanks for letting us know.

[pluginservers]

Hi Manny,

I messaged (sms) you also.
My servers are still down.
Can you please ask the techs to turn on my machines.

Thanks

Ankush Dawar

[jonathan01]

yup - still all down on dime 67

[apgraham]

Thanks for the info Manny. I still have your cell number from a hundred years ago up in NY. It was soooooo hard to resist calling you personally.

[Netlitigation]

The LAST thing I want to read here is someone from HD telling us all servers are fine... WHEN WE ARE STILL DOWN!

That is NOT what I want to see at all.

Please get the following reseller up urgently: http://72.29.65.34:2086/
Along with Dime 77, 83 and 94

And please: don't say all is fine until our sites are back up.