The Chaotic
The passage describes technological failures that demand a rapid response. Its Guardian outage anecdote also shows the value of prevention: a more realistic test dataset could have exposed the error before release.
THE IDEA
I once brought the Guardian newspaper’s travel website down for three hours, almost single-handedly. It was a simple error in retrospect; using production-like data in our tests would have caught it. I probably panicked more than the editors did, but it was my first introduction to the urgency of production bugs.
The stories of technological chaos are hard to forget. Microsoft released a chatbot on Twitter, only to find it taught how to be racist and generally offensive. Apple attempted to force everyone to listen to U2, only to find out that customers didn’t quite want it. And everyone in the UK knows about the chaos of TSB Bank’s meltdown in 2018 when it tried to perform a long-planned upgrade and locked 1.9 million people out of their accounts.
As with any chaotic situation, the trick is to act and act quickly. Of course, the software community has created tools and techniques for helping with that – some of them based on Slack. (3) Such tools can help teams coordinate a response.
When to use it
Responding to a production server outage that affects thousands of users.
Managing a PR crisis following a product failure or offensive marketing campaign.
Handling a sudden system failure that locks customers out of their accounts.
Dealing with a social media backlash from a poorly conceived or executed initiative.
FROM THEORY TO EVERYDAY LIFE
In practice
Guardian Travel Website Outage
The author recalls a simple error that brought down the Guardian’s travel website for three hours. Using production-like data in testing could have caught the error; the anecdote contrasts prevention with the urgency of responding after release.
Microsoft Racist Chatbot
Microsoft released a chatbot on Twitter that was taught to be racist and offensive, illustrating how technology can spiral into chaos quickly.
Apple U2 Album Push
Apple forced a U2 album onto customers, leading to backlash and demonstrating how even well-intentioned actions can cause chaos.
TSB Bank Meltdown 2018
A planned upgrade locked 1.9 million people out of their accounts, showing the severe consequences of chaotic system failures.
A MOMENT TO REFLECT
Put it to work.
Think of a time when a situation turned chaotic. How did you respond? What could you have done differently?
In a chaotic situation, what is the most important first step to prevent things from getting worse?
How can organizations prepare for and train teams to handle chaotic failures effectively?
CONTINUE THE CONVERSATION
Source texts
- Cynefin - Weaving Sense-Making into the Fabric of Our World ED02 (Dave Snowden Sonja Blignaut Zhen Goh)