Message provided by Bluemile Management:
At 10:30 AM EST <on 02/06/2012> this morning, Bluemile Data Center experienced a power surge from our power company followed by a loss of power. Bluemiles generator started, however, the transfer switch that transfers our feed to the generator failed. When power recovered we confirmed that the spike in power caused an issue with one of our PDU's (Power Distribution Unit). Because of the failed PDU several AC power circuits within the Data Center did not recover. All DC circuits remained stable.
During this outage the NOC lost all connectivity in to the network including phone service.
Electricians were immediately called in to repair the failed PDU and power was fully restored at 1:50PM EST. Subsequently at 3:10PM EST we took a brief power hit when the electricians repaired the transfer switch to return it to normal functionality. During the second power interruption at 3:10PM EST the UPS Batteries that feed one of our PDU Units were suspect. The interruption at 3:10PM EST did not impact all customers. Electricians have already replaced the faulty batteries and they are back online at this time.
We are still working to determine why we did not fail over to generator properly. Bluemile Network Engineering has been monitoring all the routers and switches to ensure network connectivity is back to normal and will remain onsite throughout the evening. Cloud Engineering just finished confirming our Cloud environment is back to normal.
Bluemile has been actively reviewing all network and power architectures to ensure that all precautions are taken to prevent such an occurrence in the future, and reviewing customer communication processes to ensure that we communicate with our customers in a more expeditious and consistent manner.-----End RFO-----