Hacker News new | past | comments | ask | show | jobs | submit login

When I was at Google, my team and other SRE teams around us adopted a similar approach to sensitive operations - deploys and data migrations and the like. We'd have one of us on the keyboard operating, and another looking over our shoulder. The operator would type a command and they verbally confirm the action they were going to take. Their partner would look it over and give verbal acknowledgement. We certainly still had mistakes, but I found that environment very helpful - and it was an almost necessary part of zero-blame postmortems, because every action was not one person, but equally shared between two.



I used to use check lists for this sort of critical tasks obvisly much smaller systems back then a 16 machine cluster was a big deal




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: