The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...
The fire alarm klaxon went off and the siren warning
of imminent halon gas release was screaming. We
started to panic but the data center manager shouted
over the din, "Don't worry, the halon system failed
its acceptance test last week. It's disabled and
nothing will happen."
He was half right, the primary halon system indeed
failed to discharge. But the secondary halon system
observed that the primary had conked and instantly
did its duty, which was to deal with Dire Disasters.
It had twice the capacity and six times the discharge
rate.
They don't have to blow that fast. It can be regulated.
But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.
We had it at work at a datacenter right beside offices and reception.
You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.
PS: Halon itself is of course gone but it's other less harmful gases these days.
What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).
The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
https://news.ycombinator.com/item?id=4720388 (https://web.archive.org/web/20121101194314/http://status.fog...)
https://arstechnica.com/information-technology/2012/11/how-o...
Diesel is ~7lbs/gallon. That is truly superhuman damn.
I wonder how well these "superhuman folks" were paid
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
Surprised they even had sprinklers set up. Historically, things that avoid moisture like Halon Gas [1] have been used.
[1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane
Other datacenters we built had them (which scared the shit out of me tbh). I don't remember why this one had to stay on sprinklers.
Maybe because it was in a normal office building? The gas extinguishers might be too hazardous there
Or CO2 but that's a life or death situation if there's any people still inside
Halon is not exactly fun to breathe, either.
Especially after it's been in contact with fire, the resulting products are even worse.
But it's only used in military and aviation these days anyway.
If those go off, they end up destroying hard drives (the spinning kind), due to the sound pressure....
So fire suppression and data centers don't mix too well
Ahh... the Vax War (bit hard to search for it now... gotta search for the first line). VAXen, my children, just don't belong some places.
https://www.hack.org/~mc/texts/vaxen-joke.txt
Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...
They don't have to blow that fast. It can be regulated.
But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.
Probably not safe to use on a floor that has people on it.
We had it at work at a datacenter right beside offices and reception.
You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.
PS: Halon itself is of course gone but it's other less harmful gases these days.
What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).
How is it the only datacenter?
33 Thomas St is basically one huge datacenter, that's why it doesn't have any windows. https://en.wikipedia.org/wiki/33_Thomas_Street
This was pre-Hurricane Sandy, so before 2012. I'm guessing data centers have grown since then
Work nearby, it looks just as evil in person.
Jane Street’s only NYC data center. Not the only data center in general.