"CRISIS ENGINEERING"
"TIME-TESTED TOOLS FOR TURNING CHAOS INTO CLARITY"
"AUTHOR(s)":
p7 - “If you learn only one thing from thsi book, let it be this: There is no other room, and there are no grown-ups. No one is coming. It is up to us.”
p11 - crises come in windows
p13 - “surprise forces people to accept new facts about reality”
p17 - “It is easier to adjust actions than rules, easier to shift rules than change structures, and easier to alter structures than adopt new purposes.” - Anthony Downs, Inside Bureaucracy
p22 - “machine-only systems can be changed steadily and incrementally over time, so a crisis effort probably doesn’t need to go any deeper than that.”
p23 - “As one continues to zoom out, the density of machines decreases”
p27 - “sensemaking is the way to solve a crisis”
p27 - “Daniel Kahneman and Amos Tversky (decision-making), Karl Weick (sensemaking), Donella Meadows (systems), Anthony Downs (bureau behavior), Stafford Beer (cybernetics), Nancy Leveson (safety engineering), and Joseph Heinrich (anthropology).”
p33 - “a lot of human behavior isn’t rational”
p35 - “[Karl Weick] defined sensemaking as the automatic, continuous process that determins most decisions and actions taken by a person or a group.” It’s a shared story, vision, direction or goal.
p37 - “identity”
p65 - “Sensemaking is motivated by a sense of one’s own identity” “Their actions aligned with their identity, even when reality demanded otherwise”
p75 - “introduce us to the next person we should talk to … Most of the work to make sense of the situation actually happened over lunch or dinner”
p76 - “identy the worst issue affecting the site at that instant”
p80 - “Vast Majority Sunday was declared a success, and the cable news media moved on. Notably, nothing particularly changed on that day with respect to the actual metrics we were tracking on the dashboard, but it was an emotional turning point”
p84 - tells a great bug story and a decision made.
p88 - “It is a culture where there is no possibility of being rewarded for good performance, no possibility of consequences for (most) failures, and where the only thing that can lead to punishment is to say or do something different from the rest of the agency. Government culture focuses a lot on adherence to rules and procedures, and surprisingly little on what outcomes (if any) that process achieves.”
p89 - decisions are made based on the actual goal being making it possible to win future lawsuits, not delivering a service
p98 - “Converge upon one version of the truth”
p100 - “accuracy is less important than consensus — using a single, inaccurate map trumps using several more accurate maps.”
p101 - “As you team, system maps, and data become more accurate, your action selection should become more strategic.”
- establish a war room (in person)
- map the system
- find your people
- take novel actions
- manage the story
- measure progress
- communicate
p107 - “the lobby of the Columbia Double Tree hotel”
p108 - “a sufficiently powerful overseer” … who accepts risk and “has nothing more to lose”
p115 - “It is worse to confidently declare a plan and be forced to backtrack later. Worst of all is to pretend to have a plan you aren’t sharing. No one will be fooled, and nothing more that you say will be trusted.”
p115 - “kicking people out of the center … the most successful strategy is to bore them with technical detail until they leave on their own … A bold and motivated incident lead might use the Socratic method to publicly ask the unwanted audience member a highly technical question”
p121 - “begin by tracing a single end-to-end flow”
p122 - “Maps made during normal order will be based on opinions … rather than facts”
p122 - ==“A great and terrible thing about systems thinking is that there is no obvious boundary that defines the ‘system’"==
p123 - “when to zoom in … when to zoom out” … “You will know it’s necessary to zoom out when a poorly understood external controller is causing outcomes you don’t like.”
p126 - Anthony Downs “Law of Counter-Control: ‘The greater the effort made by a … top-level official to control the behavior or subordinate officials, the greater the effort made by those subordinates to evade or counteract such control.’”
p131 - “There is a lot of value in asking two people the same question and seeing if their answers line up”
p132 - “Executives and committees hate to give visibility into how they make their decisions, lest people figure out how to manipulate them — which is, yes what you are ultimately trying to do with a map. It’s best to figure them out by observing their inputs and actions as much as you can. … it does not work well to ask people directly why they do what they do.”
p133 - “People don’t want to ask dumb questions or say they don’t know. They especially won’t do this in front of their superiors, strangers, or direct reports, and especially not in a group meeting.”
p133 - “when all else fails, use proxies”
p133 - looking at structures rather than specific contents (the example was looking at PII log files)
p138 - how do you turn it off? who can turn it off?
p139 - “when to stop updating the map”
- repeated stories
- able to predict your next meeting
- able to name the systems
- understanding normal flows
- describe how data changes based on real-world actions
- fluent in the jargon
p142 - “how to map”
- start anywhere
- consider perspectives
- basic questions
- observe directly
- ask for mysteries
- let it be messy (plausible maps have some contradictions)
- ask the same question to multiple people
- assume falsehood
p146 - finding your people
- explains in plain language
- expert in many things
- disregards formal organizational boundaries
- understands edge cases
- no orthodoxy about tech or programming language
- pragmatic decision maker
- bug chaser
- able to rapidly make decisions
p149 - “Beware of accidentally importing a preexisting clique”
p149 - normalize “graduating” out of the team once issues in their area are resolved
p150 - “Bespoke tools, ’temporary’ workarounds that are decades old, and ad hoc scripts can be viewed as inadequate or in need of replacing, but in every large system we’ve ever seen, they are essential parts of what’s going right … not a cue to shut them down or bury them in a modernization project.”
p159 - “If you reach a point where you can start and stop the machine at will, and you can make code changes, there is nothing about the machine behavior that you can’t fix.”
p162 - “The human strengths around ambiguity are enormous. The success of a complex system is determined by its adaptive capacity, and the adaptive capacity is supplied by humans.”
- this was a primary theme of the [The Origins of Efficiency](./the-origins-of-efficiency.md)
p163 - “every federal program comes with an unspoken secondary goal of being a jobs program”
p165 - “accountability sinks … the ideal process to a bureaucracy is one in which no decision can be attributed to any person.”
p165 - “Resist the management impulse to distribute rewards based on how hard people work, rather than their results.”
p189 - all work is a branch in a story web
p192 - “The crisis won’t be remembered as a success if it doesn’t have a story, and it’s not a story without an ending.“o
p197 - “limit measurements to those that align with your crisis goals”
p198 - “Gresham’s Law … Work that produces measurable outcomes tends to drive out work that produces unmeasurable outcomes. [e.g., lines of code vs. technical skill]”
p204 - “Goodhart’s Law … ‘Any meature that becomes a target ceases to be a good measure’”
p209 - “prioritize pull over push alerts”
p221 - “you cannot manufacture a crisis”
p222 - be prepared for the changes you want to see to come out of the next crisis
p226 - “It is better to end the crisis engineering effort slightly early than slightly late.”
p245 - “Complex systems are strengthened by subtraction”
p247 - “shadow processes” the undocumented processes that someone is doing and just no one knows
p266 - “most of the time most people do not want to be change agents. They want stability, money, a relatively peaceful workday, power, and the time to do other things that they care about.”
p269 - “rather than people at the outskirts being motivated to change the status quo to improve their own standing, it’s that people with the desire to challenge the status quo move themselves to these peripheral social locations.”
p295 - “Saving six minutes across twenty thousand employees gives you back zero minutes” Dan Gilbert at Quicken loans when I worked for StockX used to say “a penny saved is a penny”. Not exactly the same.