The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
Yes. And as someone who worked at Fog Creek at the time, that was a completely avoidable situation that should never have happened, and I'm absolutely gobsmacked we nearly had a repeat of it over a decade later.
Context: in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible. I was "convinced" (paid a lot) to stay on for another two years to see if Kiln would succeed. On my way out of that meeting, I commented to a coworker, "I mean, what's the chance a hurricane hits Manhattan in the next 24 months?"
100%. The chance was 100%. And the only reason people had to haul open containers of diesel up 17 flights of stairs was because no one wanted to admit that the chance wasn't 0%.
Anyway, not sure what was going on with Jane Street here, but holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
I don’t mean to glorify the poor planning or risk management, but still have a ton of respect for the people who organized the response and who did that hard work, just because it was what people needed.
My company during Sandy was operated in both NY and SF, and our experience was a bunch of us on the west coast filling in for those who were not only unable to get to the office, but without power themselves in NYC. It was all unfamiliar stuff for us (different business functions were done there, but the NYC office’s work was time critical) but did build camaraderie.
I can understand your frustration, but also let’s consider:
The chance of a hurricane within 24 months was never 100%. Looking back, it did happen, but you can’t judge future-facing decisions on post-hoc information that you didn’t have at the time.
There is a difference between “no one wanted to admit it wasn’t 0%” and “no one else thought the risk-adjusted return was positive.” When people decline to take some preventative measure, that’s usually what they’re saying implicitly.
I’ve been where you are—frustrated that you predicted and told people exactly what would happen, they didn’t listen, and you were right—many times in my career. It’s infuriating. Part of learning from it is figuring out how to get people to change, not merely figuring out how to be right.
Your comment brings back the times I've been on teams that worked double the hours a full time job should be, or spent 18 hours in a data center because a migration/data transfer was failing, and, and and.
So many times it's taken herculean effort to make things happen, usually because of poor planning and poor leadership.
Why these are viewed as "successes" or "heroic" baffles me as well.
Because it's cheaper to tell the team or company you're a hero and toss you a pizza and a day off than to hire enough people and competent leadership.
It's not a mystery, it's exploitation.
With all the business efficiencies we've made with computers as devs and IT, we should all be working 3 days weeks, instead a few people are so rich they can buy elections.
> With all the business efficiencies we've made with computers as devs and IT, we should all be working 3 days weeks, instead a few people are so rich they can buy elections.
It's never going to work like that, and you don't need any dastardly, sinister, or conspiratorial explanations for why. It's a natural side effect of the market being competitive.
When tools and technology make people more efficient, some people take that to mean they can work fewer hours and get the same job done. Which is true. But other people take that to mean they can work the same amount of hours, and get way more done. This is also true.
In a market in which two companies are competitors, and one takes the former approach and one takes the latter, which company do you think will still be around in ten or twenty years? Obviously the latter. And that pressures people in the former group to demand more output and stable working hours so they can remain competitive.
Been there, done that. 100% agree with your take.
The actual workers should be celebrated for the extra effort but the need of the extra effort should be seen as a failure. It always stops at the first part.
I'm surprised they didn't put a window washing crane on the roof as DR to simplify the raising of the diesel. The strange ass shit I've seen implemented to make the failure mode easier instead of eliminating the failure has limited my faith in our system as a whole.
Without disagreeing, I was, at the time, the lead for Kiln. I was acutely aware that any major issue with our one and only data center would result in Kiln going offline and us suffering irreparable harm, and my name would be the one tied to that, despite me fighting really hard to solve the underlying problem. Yeah, it was a different time in some ways, but I'd make that same call in the same situation again in 2026, too.
I was working at a data center in the financial district during Sandy. That was quite the experience. For whatever reason, our building was one of the last to have power turned back on. We never had to move diesel by hand fortunately. We did have a fun moment early on where one of our generators needed to have most of its load reduced for a few minutes. There were a pair of external fuel filters that needed to be changed fairly frequently. They were meant to be redundant so you could change one at a time. By the time we were able to get a tech on site, they were so dirty that when we started switching to just one the generator came very close to stalling out. After some conversation we thought it was thought that if we could drop the load enough we should be able to get away with just one dirty filter for a short period of time. The only chance we had at that was to shut down all of our cooling units. The generator was on the roof, and I was coordinating the timing over radio. You could dedinitely hear the generator wasn’t happy, but it worked. As soon as the second filter was replaced I let everyone know it was time to turn all of the cooling units back on. No one responded, and after a pause I realized that meant that probably no one had heard me and things were getting hot fast. I sprinted back inside. On the way my radio dropped, so once inside I sprinted to all of the units to tell people to turn them back on. For some reason I didn’t think about grabbing someone else’s radio. That may have been a lucky mistake though, as I’m not sure what all of those compressors turning at at the same time would have done.
I had made sure to observe the filter replacement process in case we needed to do it again ourselves. We did, and the next time someone recorded me doing it so everyone could learn and we handled most I’d the rest of the swaps from then on.
There were a lot of memorable experiences that week or so.
I would have thought that backup generators would cumulatively run for maybe a few dozen days over their lifetime. How did the fuel filters get dirty so fast?
Sandy was wild. you couldn't get computers for love or money, everything was jammed up. we ended up with whatever hardware configs had been returned and sitting in a dell warehouse just to have enough machines so for a long time after you'd have a rack of the standard build and then a random machine with a fibre channel card.
shocking how much engineering you can get done when the normal process becomes impossible and waiting isn't an option.
That one tested a lot of assumptions in planning for sure. I recall that we had a sound DR strategy, however it turns out, our main site was in Newyork and the DR site was in New Jersey to save costs and still be compliant. Good learning for sure.
> The article reminds me of when Hurricane Sandy took out a bunch of New York DCs with 3+ feet of water and people were carrying diesel up stairs to keep generators running.
Awesome, story time! I was working at a not brand new but still relatively small tech company (~$100m yearly revenue) and we were in two major datacenters, one in Las Vegas, and another in Manhattan. We were essentially telephony in the cloud, back when that was still fairly novel. At the time, most of our revenue came from handling various 1-800 campaigns, like "Ronco Knives" if you remember that, and also big money raising telethons. Our platform was active/active between the two datacenters. Of course not for a given phone call, but we could pretty easily balance traffic between both datacenters, even fully away from one of them, though as we'll see later, at some peril.
Our main networking guy was also the main datacenter person, and he'd been in NY to help set us up there. I can't remember which datacenter it was, but we had like half of a floor well above the street.
I was watching Sandy carefully, and once the odds of it hitting Manhattan were substantial, I started to press for plans to move all traffic out of the NY datacenter no less than 24 hours before landfall.
The network guy disagreed; he said that he'd been there, on site. He saw the redundant generators. The redundant network ingress and egress. The site people joked that if somehow the first floor ever flooded, the the rest of the datacenter would continue operating normally.
I noted that while all that might be true, we don't have any such assurances for all of the other equipment that'll be underwater needed to support the network.
I ended up escalating to our boss, the head of operations. He was one cube over. :) We had a vigorous discussion, and the VP agreed to pull our traffic out of NY. So we did.
Probably 75% of our capacity at the time was in LV and 25% in NY. We were running pretty hot utilization wise, but LV could, under normal circumstances, handle the whole platform by itself.
Sandy came and flooded. Generators kicked on. The network stayed up. Then the DC/networking guy got an emergency call from the datacenter.
They had plenty of fuel on site, but it turns out that the fuel inlets for the generators were...under water and could not be reached. So everything was going to go down in a couple of hours.
And it surely did, for reasons I've forgotten, in a very crappy way. Our logs indicated that power hard flapped for all of our equipment quite a few times.
We lost quite a few PSUs due to that. Yes, we definitely should have shut our equipment down cleanly. Oops.
Generators go on the 17th floor so they can vent out exhaust and heat. Pumps need to go in the basement because you can only suck a liquid up by one atmosphere, which is something like 30' for diesel, but you can push a liquid up basically as high as you want. (Also, stating the obvious, the diesel tanker trucks are gonna be street-level, so you'd need pumps there anyway if you put the actual tanks higher.) This does, are you're pointing out, kinda stink if the pump itself is flooded, but the design's common and does make sense
Probably on the 17th floor to put the generators closer to the loads and reduce transmission losses. Also reduces street level air and noise emissions (and avoids street level air intake)
With fuel storage on the same floor for a reasonable runtime, larger fuel storage in the basement and a pump (or two) to move fuel, you're well prepared for lots of things. As long as the fuel throughput on the stairs is better than the fuel use rate, then you're ok for as long as you've got people to move it. Which is ok for exceptional incidents.
You put them up there so you get a voltage boost from the gravity pushing the electricity down. Put the generators on the ground floor, and by the time the electricity reached the top, there would be a voltage drop.
No joke - have you ever tried to siphon electricity? Can't do it more than 35 feet or so. You'll be boiling off your electrons into a plasma by the time you get to that height.
> “They had their brigade of people carrying five-gallon buckets up 18 flights of stairs or whatever it was to get to their auxiliary generator. Quite a task by those guys,” Orchard said.
5 gallons of fuel at a time (hopefully not in a literal bucket) is a lot easier than managing a big drum on stairs. Unless you have a powered stair dolly, but powering your stair dolly up 17 flights of stairs seems like a challenge too. Team lifting a drum up one flight of stairs feels managable, doing it up 17 flights multiple times is a hard no.
5 gallons is a standard size of literal buckets, they can have lids but I might prefer carrying a bucket of diesel up 18 flights of stairs with no lid over trying to get one of those goddamn lids off.
Apparently it was a 30-person 5 gallon bucket brigade of customers, contractors, and volunteers all lining up to help fill tanks because of interconnection issues between the supplier they got and where they needed to fuel the tanks up with.
"... neither we nor the fuel company had the proper coupling to tie into that existing infrastructure."
Sometimes "DR" in your plans stands for "didn't realize".
As a bonus though? They were able to in fact get ahead of the fuel burn, and the crew managed an eight hours' head back up on the tanks.
In the article he talks about living within a few minutes walk to work and telling too many people so they made him the emergency office guy. Same thing happened to me, foolishly told everyone hey I can see my apartment from here so I would get panicked calls to run in and shut down servers because the information security guy got a tip of an impending exploit about to be released on a holiday weekend with no way to do it remotely. Everyone else lived a ferry ride or commuter train away.
Not the person you asked, but I've learned that if you're not careful about what you tell your employers and/or coworkers that it will almost invariably come back to haunt you in the form of unpaid labor.
It took me an embarrassingly long time to realize that (my 30's), and now I keep my conversations with my coworkers very utilitarian and divulge minimal information about myself. I try not to be a dick or anything, I'm polite enough, but pretty much everything I talk about is either directly relevant to work and/or so surface level as to border on "vapid".
ETA:
Just as an example, I used to have really bad sleep issues in my 20's that would make it so I wouldn't be able to get to sleep until around 4am, if at all. I made the mistake of bringing that up at work to my manager. Two weeks later I got a phone call at 3am because there was something he wanted to look at and "he knew I would be up". I was up, but I didn't really have much desire to get my laptop out and do work. I liked that job at the time and I liked my manager, so I did it anyway, and it wasn't until years later did I think about that incident and how utterly weird it was that I was sort of being punished for my insomnia.
My sleep issues have gotten a lot better now, so if someone calls me at 3am I probably won't hear it because I'll be asleep, but I remember that incident now and have it constantly spinning in my brain when I chat with coworkers.
The problem here is not divulging info about yourself, its not being able to say no to work. I will happily draw a line where my responsiblities end and if anybody asks me to go over that I will ask for a role review. Too many people get walked over at work, stand up for yourself and say no!
Still, I do think there's merit in keeping your personal life to yourself and not telling your coworkers more than they need to know. I'm not saying you should be stoic or a dick or anything; by all means engage in conversations and be nice. I'm just saying that it's important to have boundaries.
I tend to be someone who overshares a lot (hence my enormous word count on Hacker News), so I just try and keep this in mind.
Fair response. I guess it also depends on the working environment, the type of person you are, and whether your coworkers are more or less like you as a person etc.
I'm fine sharing (some) things about myself with coworkers, but I'm also in a position where if someone says, "So you can be our on-call guy for on-prem servers?" I can say "Absolutely not without a really big fucking raise, and I am not kidding, if you call me at 3am to come in, I will hang up on you."
I've learned that life is what you make of it. If you let yourself be used - others will use you. But if you showed your worth, but also showed others that you know your worth - things are different.
(otherwise where I work, I also avoid talking about politics for instance, but this is a different theme)
I don't really talk about politics at work either, but that's broadly because I live in a very progressive area where pretty much everyone has the same opinions that I do so it would be a pretty boring echo chamber.
Odd that they didn't have at minimum 1) a physical Emergency Power Off (EPO) button inside the datacenter, such that one button could de-energize everything -- usually required as part of building a datacenter due to code variance for the "low voltage" (<4000V in this context...) wiring. 2) EPO interlink to the smoke alarms, such that it would de-energize equipment automatically (maybe with delay) if smoke detected.
Interesting. The Sequence of Operations we write for the data halls I work on specifically do not shut down the CRACs on smoke alarm for this reason. The International Mechanical Code (IMC) specifically allows this where a air handler over 2000 cfm serves only one space because it's recognized that it will not spread smoke around a building. 2000 cfm is the threshold where you typically have to install smoke detectors, and typically they do shut down on smoke detection.
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
Great post. I hope a big paycheck was given to the author for saving the DC and the company from disaster.
Many many years ago someone told me that your data is not safe if is not at least in 3 very different physical places. Today, I would probably add the requirement to be on different providers as well.
> every minute, ask every server in each location directly for the temperature from its onboard sensors, take the average, and if it’s elevated, blast an alert out to everyone on the systems team to check for datacenter overheat.
I've once done the opposite: the office "server room" (a small room with three racks, two of them for the patch panels, one for the servers and switches) was cooled by a mini-split which was prone to locking up with the compressor enabled (the relay which controls the compressor got stuck) and freezing, which then stopped the air from circulating due to all the ice. I wrote a script which ran on one of the servers which read the ambient temperature through IPMI to detect when the room was too cold, email a warning, and power off (using ssh) the neighboring servers and then itself.
I wonder why the answer was to turn off the servers instead of turn their cooling back on. Would it not be able to stabilize the temperature fast enough?
The real answer is probably lost in the sands of time but even at 200 people we still had different teams with overlapping responsibilities. It's possible I was the only person reachable and close enough after hours, and while I didn't know where the CRAC unit controls were I did know how to rip power cords out.
They'd be shutdown by a dry contact that may variously be coming from the VESDA (Very Early Smoke Detection Apparatus), or an internal smoke detector, or a duct mounted smoke detector. Not as simple as flipping an obvious switch on a front panel.
I specifically remember a TV commercial many years ago (15-20 maybe) that was trying to show the sophistication of the company with an anecdote of how when looking at some other company as a potential investment, considered if they used sprinklers in their data centers and determined that they did not. I could have sworn the company advertising was Jane Street.
That's how the server room at my school worked. They said to get out of there if the alarm went off because the room would fill with some inert gas and you'd suffocate.
This would be the one time I get to hit the EPO button (which this DC seemed shockingly too ghetto to have) for our breakers and it not being a resume-generating event.
I’m surprised by the pulling cords by hand part. Did they really not have an emergency power off wired up to a shunt trip or some other device to efficiently cut the power? Isn’t this mandatory, even? Especially in a datacenter in an office building?
Nitpick, but the headline is not confusing at all, even though a couple of people are apparently confused. I suspect that is the fault of a hasty read through, not the headline itself.
There's tons of substitutes, the ban on Halon wouldn't be why Jane St is using water.
Many people (like me) use "halon" when they are talking about general inert gas based fire suppression as there's not really a good short word for them otherwise.
Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...
The fire alarm klaxon went off and the siren warning
of imminent halon gas release was screaming. We
started to panic but the data center manager shouted
over the din, "Don't worry, the halon system failed
its acceptance test last week. It's disabled and
nothing will happen."
He was half right, the primary halon system indeed
failed to discharge. But the secondary halon system
observed that the primary had conked and instantly
did its duty, which was to deal with Dire Disasters.
It had twice the capacity and six times the discharge
rate.
They don't have to blow that fast. It can be regulated.
But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.
We had it at work at a datacenter right beside offices and reception.
You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.
PS: Halon itself is of course gone but it's other less harmful gases these days.
What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).
“The superhuman folks at the data center are hauling 55 gallon drums of diesel fuel up 17 flights of stairs. Assuming they're able to sustain a rate of transfer that exceeds the rate of consumption, the data center should run until electricity is restored. The latest estimate from Con Edison for lower Manhattan is 4 days. (The New York Stock Exchange will be open tomorrow, also running on generator power.)”
https://news.ycombinator.com/item?id=4720388 (https://web.archive.org/web/20121101194314/http://status.fog...)
https://arstechnica.com/information-technology/2012/11/how-o...
Context: in 2012, I literally quit Fog Creek, largely over the fact that we had no backup strategy and I was told that a backup strategy was not feasible. I was "convinced" (paid a lot) to stay on for another two years to see if Kiln would succeed. On my way out of that meeting, I commented to a coworker, "I mean, what's the chance a hurricane hits Manhattan in the next 24 months?"
100%. The chance was 100%. And the only reason people had to haul open containers of diesel up 17 flights of stairs was because no one wanted to admit that the chance wasn't 0%.
Anyway, not sure what was going on with Jane Street here, but holding up that Fog Creek/Trello/SquareSpace story as a positive always drives me somewhat nuts. It lionizes a failure.
My company during Sandy was operated in both NY and SF, and our experience was a bunch of us on the west coast filling in for those who were not only unable to get to the office, but without power themselves in NYC. It was all unfamiliar stuff for us (different business functions were done there, but the NYC office’s work was time critical) but did build camaraderie.
The chance of a hurricane within 24 months was never 100%. Looking back, it did happen, but you can’t judge future-facing decisions on post-hoc information that you didn’t have at the time.
There is a difference between “no one wanted to admit it wasn’t 0%” and “no one else thought the risk-adjusted return was positive.” When people decline to take some preventative measure, that’s usually what they’re saying implicitly.
I’ve been where you are—frustrated that you predicted and told people exactly what would happen, they didn’t listen, and you were right—many times in my career. It’s infuriating. Part of learning from it is figuring out how to get people to change, not merely figuring out how to be right.
So many times it's taken herculean effort to make things happen, usually because of poor planning and poor leadership.
Why these are viewed as "successes" or "heroic" baffles me as well.
It's not a mystery, it's exploitation.
With all the business efficiencies we've made with computers as devs and IT, we should all be working 3 days weeks, instead a few people are so rich they can buy elections.
It's never going to work like that, and you don't need any dastardly, sinister, or conspiratorial explanations for why. It's a natural side effect of the market being competitive.
When tools and technology make people more efficient, some people take that to mean they can work fewer hours and get the same job done. Which is true. But other people take that to mean they can work the same amount of hours, and get way more done. This is also true.
In a market in which two companies are competitors, and one takes the former approach and one takes the latter, which company do you think will still be around in ten or twenty years? Obviously the latter. And that pressures people in the former group to demand more output and stable working hours so they can remain competitive.
Quitting a job because of this is so funny to me. 2012 was truly a different time.
I had made sure to observe the filter replacement process in case we needed to do it again ourselves. We did, and the next time someone recorded me doing it so everyone could learn and we handled most I’d the rest of the swaps from then on.
There were a lot of memorable experiences that week or so.
https://x.com/BrettHarrison/status/2117636364304892206
shocking how much engineering you can get done when the normal process becomes impossible and waiting isn't an option.
Awesome, story time! I was working at a not brand new but still relatively small tech company (~$100m yearly revenue) and we were in two major datacenters, one in Las Vegas, and another in Manhattan. We were essentially telephony in the cloud, back when that was still fairly novel. At the time, most of our revenue came from handling various 1-800 campaigns, like "Ronco Knives" if you remember that, and also big money raising telethons. Our platform was active/active between the two datacenters. Of course not for a given phone call, but we could pretty easily balance traffic between both datacenters, even fully away from one of them, though as we'll see later, at some peril.
Our main networking guy was also the main datacenter person, and he'd been in NY to help set us up there. I can't remember which datacenter it was, but we had like half of a floor well above the street.
I was watching Sandy carefully, and once the odds of it hitting Manhattan were substantial, I started to press for plans to move all traffic out of the NY datacenter no less than 24 hours before landfall.
The network guy disagreed; he said that he'd been there, on site. He saw the redundant generators. The redundant network ingress and egress. The site people joked that if somehow the first floor ever flooded, the the rest of the datacenter would continue operating normally.
I noted that while all that might be true, we don't have any such assurances for all of the other equipment that'll be underwater needed to support the network.
I ended up escalating to our boss, the head of operations. He was one cube over. :) We had a vigorous discussion, and the VP agreed to pull our traffic out of NY. So we did.
Probably 75% of our capacity at the time was in LV and 25% in NY. We were running pretty hot utilization wise, but LV could, under normal circumstances, handle the whole platform by itself.
Sandy came and flooded. Generators kicked on. The network stayed up. Then the DC/networking guy got an emergency call from the datacenter.
They had plenty of fuel on site, but it turns out that the fuel inlets for the generators were...under water and could not be reached. So everything was going to go down in a couple of hours.
And it surely did, for reasons I've forgotten, in a very crappy way. Our logs indicated that power hard flapped for all of our equipment quite a few times.
We lost quite a few PSUs due to that. Yes, we definitely should have shut our equipment down cleanly. Oops.
With fuel storage on the same floor for a reasonable runtime, larger fuel storage in the basement and a pump (or two) to move fuel, you're well prepared for lots of things. As long as the fuel throughput on the stairs is better than the fuel use rate, then you're ok for as long as you've got people to move it. Which is ok for exceptional incidents.
You get out of it what you put into it. ;)
> “They had their brigade of people carrying five-gallon buckets up 18 flights of stairs or whatever it was to get to their auxiliary generator. Quite a task by those guys,” Orchard said.
5 gallons of fuel at a time (hopefully not in a literal bucket) is a lot easier than managing a big drum on stairs. Unless you have a powered stair dolly, but powering your stair dolly up 17 flights of stairs seems like a challenge too. Team lifting a drum up one flight of stairs feels managable, doing it up 17 flights multiple times is a hard no.
Bucket lids are a PITA, but so is spilling diesel on the stairs.
Apparently it was a 30-person 5 gallon bucket brigade of customers, contractors, and volunteers all lining up to help fill tanks because of interconnection issues between the supplier they got and where they needed to fuel the tanks up with.
"... neither we nor the fuel company had the proper coupling to tie into that existing infrastructure."
Sometimes "DR" in your plans stands for "didn't realize".
As a bonus though? They were able to in fact get ahead of the fuel burn, and the crew managed an eight hours' head back up on the tanks.
It took me an embarrassingly long time to realize that (my 30's), and now I keep my conversations with my coworkers very utilitarian and divulge minimal information about myself. I try not to be a dick or anything, I'm polite enough, but pretty much everything I talk about is either directly relevant to work and/or so surface level as to border on "vapid".
ETA:
Just as an example, I used to have really bad sleep issues in my 20's that would make it so I wouldn't be able to get to sleep until around 4am, if at all. I made the mistake of bringing that up at work to my manager. Two weeks later I got a phone call at 3am because there was something he wanted to look at and "he knew I would be up". I was up, but I didn't really have much desire to get my laptop out and do work. I liked that job at the time and I liked my manager, so I did it anyway, and it wasn't until years later did I think about that incident and how utterly weird it was that I was sort of being punished for my insomnia.
My sleep issues have gotten a lot better now, so if someone calls me at 3am I probably won't hear it because I'll be asleep, but I remember that incident now and have it constantly spinning in my brain when I chat with coworkers.
Still, I do think there's merit in keeping your personal life to yourself and not telling your coworkers more than they need to know. I'm not saying you should be stoic or a dick or anything; by all means engage in conversations and be nice. I'm just saying that it's important to have boundaries.
I tend to be someone who overshares a lot (hence my enormous word count on Hacker News), so I just try and keep this in mind.
(otherwise where I work, I also avoid talking about politics for instance, but this is a different theme)
A) because picturing at my current work place everybody learning of all my opinions at once and the resulting drama
B) I am in progressive spaces quite often, but they are far from an echo chamber with everyone having the same opinions either ..
The CRAC won't feed a fire oxygen as it's just recirculating the same oxygen depleted air.
https://signalsandthreads.com/the-thermodynamics-of-trading/
Many many years ago someone told me that your data is not safe if is not at least in 3 very different physical places. Today, I would probably add the requirement to be on different providers as well.
I've once done the opposite: the office "server room" (a small room with three racks, two of them for the patch panels, one for the servers and switches) was cooled by a mini-split which was prone to locking up with the compressor enabled (the relay which controls the compressor got stuck) and freezing, which then stopped the air from circulating due to all the ice. I wrote a script which ran on one of the servers which read the ambient temperature through IPMI to detect when the room was too cold, email a warning, and power off (using ssh) the neighboring servers and then itself.
Why not just turn the CRAC back on?
> you don’t trust the automation unless there’s a big red-painted, mechanically connected lever you can pull to turn it off
Seems like this is one of those cases where you want to override the automation.
33 Thomas St is basically one huge datacenter, that's why it doesn't have any windows. https://en.wikipedia.org/wiki/33_Thomas_Street
Nitpick, but the headline is not confusing at all, even though a couple of people are apparently confused. I suspect that is the fault of a hasty read through, not the headline itself.
[1] - https://en.wikipedia.org/wiki/Bromotrifluoromethane
Many people (like me) use "halon" when they are talking about general inert gas based fire suppression as there's not really a good short word for them otherwise.
So fire suppression and data centers don't mix too well
https://www.hack.org/~mc/texts/vaxen-joke.txt
Realizing that I'm spoiling some of it... but we're here, so some of it is inherently spoiled...
But yes, I've seen low-O2 installations also. These have other issues though, like no protection when humans need to be in there for maintenance. And the need to be very very careful when reducing it again, making sure everyone is out. It happened to my colleague once that this went wrong and while it's not low enough that you'll die, he wasn't exactly feeling great for the next few hours.
But it's only used in military and aviation these days anyway.
You just have to reinforce the ceiling panels and the windows so they don't get blown out, but it's not really an issue otherwise. It was all certified and regularly checked. Water is indeed a bit counterproductive as it's not really much better than fire for the equipment in there.
PS: Halon itself is of course gone but it's other less harmful gases these days.
What also happens is low-oxygen environments (they only raise the O2 when maintenance is being done).
I smell AI