As many of you are aware, over the past 18 hours there has been an unprecedented amount of flooding and damage to the NYC and surrounding areas. The flooding has had a major impact on the city’s infrastructure including the electrical grid, public transportation and accessibility to portions of Manhattan.
At approximately 7 PM ET last night, ConEd began proactively turning down portions of the power grid in preparation for the expected flood of saltwater from Sandy’s storm surge. As they did so, many facilities, including Internap’s data centers at 111 8th Ave. and 75 Broad, transferred to back-up generator power as designed.
Around 9 PM the storm surge, combined with high tide, caused flooding to many portions of lower Manhattan, including in and around our data center at 75 Broad. At the peak there was three feet of flood water in the lobby at the site, causing both basement floors to be inaccessible. As a result of the flooding, both our redundant fuel pumps and our generator fuel tank were compromised and shut down. The system continued to run until all fuel within the secondary feeder tanks were exhausted and our facility lost power.
We are working as quickly as possible to implement a workaround for the fuel system that will allow us to bring the generator farm back into operation. It is unclear how long it will take ConEd to restore utility power to the site, but we are preparing for the possibility of remaining on generator power for many days.
Our 111 8th Ave. data center continues to run on generator power successfully. We have enough fuel to last several days and expect fuel delivery to the site will be possible prior to depleting on-site reserves. We continue to see many network issues throughout the region, with many vendors losing either power or facilities due to flooding or the like. All of our network points are currently working as expected and have ample capacity to deal with the other provider outages.
If you are having trouble or have questions regarding any of your Internap services please contact our NOC and/or support teams as follows:
Network Operations Center
- noc@internap.com
- 877.843.4662
Agile Hosting
- agile-support@internap.com
- 877.843.4662, ext. 3
Managed Hosting
- ms-support@internap.com
- 877.843.4662
Lastly, I want to thank so many of our customers for being understanding of these very difficult circumstances. Our support teams are working around the clock and sparing no efforts to resolve any outstanding issues and we will continue to update you as information is available.
All of you, your families, your friends and the people of the impacted regions are in our thoughts.
