In DevOps (and Lean, TPS) the more advanced form of this is the Poka-Yoke (https://en.wikipedia.org/wiki/Poka-yoke). Poka-yokes don't just add safety, they also guide the human away from making a mistake.
The canonical example is the automatic shift knob in a car. The shift knob is designed to 1) prevent you from accidentally shifting all the way back into reverse without pressing the shift button, and 2) prevents you from leaving park or neutral without depressing the brake pedal. This way you don't damage the drivetrain or accidentally cause the car to roll forward/backward.
Poka-yoke is a form of defensive design (https://en.wikipedia.org/wiki/Defensive_design). For a beautiful example of defensive design, see the average electric kettle. If water boils over the top it won't short the device, if it boils dry it'll stop operating, the handle and body are plastic to prevent burning yourself, the handle is ergonomic to make carrying 1.5L of sloshing boiling water not cause you to spill it, the cord is detached from the kettle so you don't yank the cord and spill the boiling water, the switches are located on the bottom away from hot steam, and the lids usually lock while in operation, again to prevent damage from spillage or steam. It's the simplest and safest possible way to boil water, and it's $20.
My favorite example of poka-yoke is when the pieces and hardware in build-it-yourself furniture kits won't fit anywhere except the correct places: two screws only have the same width if they're interchangeable, wood bars refuse to go in unless facing the right direction, etc.
There's a great piece of software called "molly-guard", which intercepts calls to "poweroff" and "reboot" and similar. It checks if it's being invoked via an SSH session, and if so, it asks you to type the name of the system you're shutting down. That way, you never accidentally shut down a remote server when you meant to shut down your own system (or a different server).
I once accidentally rebooted the reverse proxy for all our production traffic. We got some very quiet two minutes while it came back up.
After that we installed molly-guard with a check for the number of active connections. Made it painless to reboot standby proxies and difficult to reboot active ones.
(We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)
I like telling junior hires about this incident because it teaches them that (a) anyone can make mistakes, (b) even serious mistakes usually aren't that dangerous, (c) you can learn a lot from mistakes with the right mindset, (d) we cannot prevent mistakes but with the right system design we can reduce their consequences.
> (We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)
It's a great opportunity to share knowledge and techniques and I very much recommend doing so. It's an important way to get people familiar and comfortable with what the documentation says. Or, it's less scary to failover a database or an archiving clutser while the DBA or an archive admin is in a call with you.
Also reminds me of an entirely funny culture shock for a new team member, who was on a team with a much worse work culture and mutual respect beforehand. Just 2-3 months after she joined, we had a major outage and various components and clusters needed to be checked and put back on track. For these things, we do exactly this pilot/copilot structure if changes to the system must go right.
Except, during this huge outage, two people were sick, two guys had a sick kid, one guy was on a boat on the northern sea, one guy was in Finland and it was down to 3 of the regulars and the junior. Wonderful. So we shoved her the documentation for one of the procedures and made her the copilot of her mentor and then we got to work, just calmly talking through the situation.
Until she said "Wait". And some combined 40 - 50 years of experience stopped on a dime. There was a bit of confusion of how much that word weighed in the team, but she did correctly flag an inaccuracy in procedure we had to adress, which saved a few minutes of rework.
I was using my company dev machine via Windows RDP remotely during Covid and installed Glasswire which by default blocks all traffic so I lost access. No one was there to uninstall it so I continued development in my personal machine.
Another fun one is disabling the network interface on a remote server. An acquaintance did that by mistake on a cloud VM running some core services, and the cloud provider had no virtual console for some reason. Ended up having to write off the VM and restore from backup. Fun day at the office.
Long ago, I succeeded once to cut my own access through SSH to a remote server, after some firewall changes. That of course has required a long trip to the server, for physical access.
However that was good, because after that I have always been extra careful at any changes that could affect the firewall in any way. (That is not restricted to changes in firewall rules, because there are systems where the versions of the firewall program and of the kernel must be correlated, so an inconsistent update may make the firewall revert to its default state of denying all connections.)
I am confused by the second guy who was curious and punched the plastic lid… it says you have to hold the button down for 30 seconds, how did that happen?
Samsung learned about molly guards the hard way - recall of millions of products after accidental fires from people/pets activating the front-panel dials.
Luckily, there’s an easy solution recently devised that can prevent this safety hazard in homes across America, Samsung said. Customers concerned about unintentional activations can request free knob locks and covers that Samsung confirmed made it much harder to accidentally turn on the stove.
During the meeting, the CPSC shared data showing that across 338 incidents between January 1, 2018, and May 30, 2024, stoves from “ten specific manufacturers” were involved in fires causing 31 injuries and two deaths. Additionally, the CPSC had recorded “two other fatal incidents where a range was accidentally turned on when a knob was bumped, but the manufacturer is unknown.”
...Companies said the CPSC data would help them “fully understand the issues” and “make sure that reasonable and foreseeable circumstances would be addressed” without impacting compliance with the Americans with Disabilities Act.
After mentioning this article to relatives, one said they had nixed buying one product because of the front dials. Then we heard from a relative in another city who bought a house due to a newborn baby - one of the additional purchases was a oven/stove/range with front panel dials.
That's wild. In Europe, we generally only have front-facing dials, but either they have a built-in push to recess function, or they require some force to turn from their off position, and for the most part heat and mode are also two separate knobs where you'd need to turn both to engage.
> There is no worse feeling for a programmer than waking up, walking up to the machine that was supposed to work through the night, and seeing it did absolutely nothing, stupidly waiting for hours for a response to a question that didn't even matter.
No, there's one worse feeling. Walking up to the machine that was supposed to work throughout the night, and seeing it had a surprise update that rebooted the system.
My dad worked for Sperry Univac. For a while he was working on a ground-support trailer for the Sergeant surface-to-surface missile. He went in to work one Saturday for some kind of a major test. For some reason, he brought my mother and I along (maybe to give my mom a break). So this four-year-old (yours truly) goes into the trailer, and sees this bright red button...
It was not the launch button. It was the emergency shutdown button, which would have cost them an hour to bring the trailer back online. Someone stopped me before I actually pushed it, but still, this did not make me popular. What I remember from that day is actually the parking lot, because I spent far more time in the parking lot than in the trailer.
I feel like modern tv remotes are the opposite of this principle. It is often the case that almost every single button will when pushed in some way interrupt the current program, often jumping out to a different menu or changing to a different program or something. It makes handling the remote or trying to change the volume a fraught experience.
OMG you hit my third rail when it comes to the brain-dead-designed Apple TV remote. After using one for many years I STILL press the wrong button many times every day, and in the dark, since the buttons are NOT backlit, I routinely press an unintended button. I think the user interface designer was promoted to create the Vision Pro UI/UX which is even more dreadful.
I once was a communications contractor for the major NJ power utility. One of their long time field techs (let’s just refer to him as Mr. T) was giving a tour of a substation that was built from the looks of it in the 50s. I have, you see, this bad habit of leaning on things… well Mr. T, without missing a beat, slid his forearm between my hip and a faded green Bakelite knob, the kind that goes in and out rather than twisting. He informed me that if I had leaned any further I would have shut off half of Newark.
Seeing long presses implemented for those intermittent and irreversible actions in games is something I‘ve always appreciated. I often end up making errant inputs, especially on keyboards.
A guard I often make for myself is removing/disabling the delete key on my keyboard, and setting FN+Backspace to Delete with whatever control software is involved. I often then repurpose the delete key location to F2, which is typically used to “Edit” a spreadsheet cell or file name.
I think that may be a combination of (IMHO unfortunate) factors:
* Yes, on some systems rm is aliased to rm -i by default.
* Some scripts will use rm -f because normal rm returns an error if the target already doesn't exist but -f doesn't care.
* Finally, sometimes files are just ... I think it's being marked read-only that does it? I've hit this while trying to rm a git checkout; you actually do need to add -f sometimes to succeed. So if you just add -f then it'll always work.
Such pop-ups should never automatically get focused. The increase of them was why I switched away from Windows many years ago, and why I like to root my Android devices. It baffles me that focus-stealing notifications cannot be turned off in most OEM Androids.
I do wish those were a thing on flat touch sensitive induction cooktops! (For all those pesky water droplets causing the cooktop to error out and turning itself off)
Just please don't start adding molly-guards to your software. The concept only makes sense in the physical world, e.g. where the "important button", that you might never have to press, needs to be in reach all the time. In software, there are better solutions.
my favorite Debian package is Mollyguard so when you shut down a server remotely via SSH it just checks the second time to make sure you really wanted to shut down that server and not your laptop.
"Are you sure?" type guards are not suitable for actions which the user does regularly. If a user repeats this action regularly, they quickly automate the thought process (i.e. don't give it any thought anymore) and it becomes useless.
I agree. Fortunately, molly-guard the software can be configured with automated checks to allow safe actions (e.g. shutting down servers that don't receive significant traffic) without pestering the user.
This means a properly configured mollly-guard is invisible for routine actions but kicks in only when a genuine mistake is suspected because the operation would cause some sort of meaningful loss. That way, users aren't trained to ignore it.
>> At 08:56 a ‘Trade Limit Warning’ pop-up alert appeared within PTE. This presented the trader with 711 warning messages, consisting of hard block and soft block messages, listed in a single alert where only the first 18 lines of alerts were immediately visible unless the person who received the alert scrolled down. The trader did not appreciate their inputting error and overrode all of the soft warnings in the pop-up.
> You get 711 alerts, you only see 18 of them, you are like “ehh 18 alerts is pretty much the normal number,” you override them all without reading.
It's not nitpicking. The nature of the interruption being different is material. I've lost files to automatically answering yes to rm -i y/n confirm. Typing the hostname itself is different enough to get me, at least to stop and go wait, hold on. And snap me out of doing the wrong one. Especially an SSH gateway machine.
The canonical example is the automatic shift knob in a car. The shift knob is designed to 1) prevent you from accidentally shifting all the way back into reverse without pressing the shift button, and 2) prevents you from leaving park or neutral without depressing the brake pedal. This way you don't damage the drivetrain or accidentally cause the car to roll forward/backward.
Poka-yoke is a form of defensive design (https://en.wikipedia.org/wiki/Defensive_design). For a beautiful example of defensive design, see the average electric kettle. If water boils over the top it won't short the device, if it boils dry it'll stop operating, the handle and body are plastic to prevent burning yourself, the handle is ergonomic to make carrying 1.5L of sloshing boiling water not cause you to spill it, the cord is detached from the kettle so you don't yank the cord and spill the boiling water, the switches are located on the bottom away from hot steam, and the lids usually lock while in operation, again to prevent damage from spillage or steam. It's the simplest and safest possible way to boil water, and it's $20.
After that we installed molly-guard with a check for the number of active connections. Made it painless to reboot standby proxies and difficult to reboot active ones.
(We also instituted pairing on production proxy maintenance. I'm not a fan of pair programming but pair maintenance is great.)
I like telling junior hires about this incident because it teaches them that (a) anyone can make mistakes, (b) even serious mistakes usually aren't that dangerous, (c) you can learn a lot from mistakes with the right mindset, (d) we cannot prevent mistakes but with the right system design we can reduce their consequences.
It's a great opportunity to share knowledge and techniques and I very much recommend doing so. It's an important way to get people familiar and comfortable with what the documentation says. Or, it's less scary to failover a database or an archiving clutser while the DBA or an archive admin is in a call with you.
Also reminds me of an entirely funny culture shock for a new team member, who was on a team with a much worse work culture and mutual respect beforehand. Just 2-3 months after she joined, we had a major outage and various components and clusters needed to be checked and put back on track. For these things, we do exactly this pilot/copilot structure if changes to the system must go right.
Except, during this huge outage, two people were sick, two guys had a sick kid, one guy was on a boat on the northern sea, one guy was in Finland and it was down to 3 of the regulars and the junior. Wonderful. So we shoved her the documentation for one of the procedures and made her the copilot of her mentor and then we got to work, just calmly talking through the situation.
Until she said "Wait". And some combined 40 - 50 years of experience stopped on a dime. There was a bit of confusion of how much that word weighed in the team, but she did correctly flag an inaccuracy in procedure we had to adress, which saved a few minutes of rework.
However that was good, because after that I have always been extra careful at any changes that could affect the firewall in any way. (That is not restricted to changes in firewall rules, because there are systems where the versions of the firewall program and of the kernel must be correlated, so an inconsistent update may make the firewall revert to its default state of denying all connections.)
https://entropicthoughts.com/locking-yourself-out-with-firew...
see https://arstechnica.com/tech-policy/2024/08/samsung-recalls-...
After mentioning this article to relatives, one said they had nixed buying one product because of the front dials. Then we heard from a relative in another city who bought a house due to a newborn baby - one of the additional purchases was a oven/stove/range with front panel dials.I never heard of any related injuries over here.
No, there's one worse feeling. Walking up to the machine that was supposed to work throughout the night, and seeing it had a surprise update that rebooted the system.
One of my favorite things about ditching Windows.
https://en.wikipedia.org/wiki/Ed_Krol
My dad worked for Sperry Univac. For a while he was working on a ground-support trailer for the Sergeant surface-to-surface missile. He went in to work one Saturday for some kind of a major test. For some reason, he brought my mother and I along (maybe to give my mom a break). So this four-year-old (yours truly) goes into the trailer, and sees this bright red button...
It was not the launch button. It was the emergency shutdown button, which would have cost them an hour to bring the trailer back online. Someone stopped me before I actually pushed it, but still, this did not make me popular. What I remember from that day is actually the parking lot, because I spent far more time in the parking lot than in the trailer.
Also, perhaps `rm` should be molly guarded to move things to the trash on all systems by default, and delete only if forced to by a flag.
Note: I’d have expected Molly to be a cat, because they tend to be pretty good at disrupting things in my experience.
Not all systems, but some (RHEL, I think?) default alias rm='rm -i', yes
A guard I often make for myself is removing/disabling the delete key on my keyboard, and setting FN+Backspace to Delete with whatever control software is involved. I often then repurpose the delete key location to F2, which is typically used to “Edit” a spreadsheet cell or file name.
To me, a button that forces me to wait for an unknown and indeterminate period of time before functioning is a FAIL.
* Yes, on some systems rm is aliased to rm -i by default.
* Some scripts will use rm -f because normal rm returns an error if the target already doesn't exist but -f doesn't care.
* Finally, sometimes files are just ... I think it's being marked read-only that does it? I've hit this while trying to rm a git checkout; you actually do need to add -f sometimes to succeed. So if you just add -f then it'll always work.
This means a properly configured mollly-guard is invisible for routine actions but kicks in only when a genuine mistake is suspected because the operation would cause some sort of meaningful loss. That way, users aren't trained to ignore it.
That's clever. This is what I meant when I wrote, that software allows for better solutions.
>> At 08:56 a ‘Trade Limit Warning’ pop-up alert appeared within PTE. This presented the trader with 711 warning messages, consisting of hard block and soft block messages, listed in a single alert where only the first 18 lines of alerts were immediately visible unless the person who received the alert scrolled down. The trader did not appreciate their inputting error and overrode all of the soft warnings in the pop-up.
> You get 711 alerts, you only see 18 of them, you are like “ehh 18 alerts is pretty much the normal number,” you override them all without reading.
I discussed this also here:
https://news.ycombinator.com/item?id=46845740