“Where there is judgment, there is noise – and more of it than you think.” – Kahneman, et al Noise: A Flaw in Human Judgment
The city I live in is home to a very old, very beautiful, and very famous cathedral. It’s been a hotspot for tourists since the days of Chaucer (of course they weren’t called tourists then) and still plays an integral role in the community.
No matter where you are in the city, you can see that distinctive spire above everything. If you managed to get turned out (not as difficult as you might think with many small, medieval streets), that spire will lead you straight back to the center of town. It’s pervasive and omnipresent in such a way that – ultimately – leads to it being utterly ignored.
The main thing you need to know about this cathedral, though, is that it has bells. Loud bells, and many of them. And these bells toll. A lot. Like a lot. Not that briefly pleasant trill of bells you might get on the hour or quarter-hour, either. Some days, I’d swear the bells never stop ringing at all.
The thing is, much like the spire, I never paid much attention to the bells – until they stopped. For the year+ of varying levels of lockdown during the pandemic, the bells remained silent. Lockdown ended and things started reopening, including the cathedral and the cacophonic celebration of every pigeon coo by those accursed bells.
Yeah, it’s become a bit of a thing.
This, dear reader, is an example of noise – both literally and figuratively. The cathedral has always been there – unobtrusive, in the background, subtly influencing my actions and habits without any conscious awareness on my part. It shapes the way I move through the city, my awareness of time, my mood, my memories; but, before lockdown, if anyone asked me if it had any impact on my life, I probably would’ve shrugged and answered, “Not really.”
Noise is everywhere, and it affects every decision we make – especially because we aren’t even aware of it. We all know to check our biases when it comes to important decisions, but how often do you check your noise?
At Process Street, we take good decision hygiene very seriously, so this post will examine the idea of noise presented by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein in their book, Noise: A Flaw in Human Judgment, and use those ideas in our very own Noise Audit Workflow.
Feel free to skip ahead for the workflow, or stick with me as I discuss the theory of noise:
- What do we mean by “noise”?
- Noise in action: Be quiet! I’m trying to think.
- Reduce noise by focusing on the process
- Check your noise with the Process Street Noise Audit Workflow
Let’s make some noise!
What do we mean by “noise”?
“Noise is the variability where you don’t want it.” – Daniel Kahneman, Nobel Prize winner
Errors in judgment systems are caused by two things: bias and noise. We’ve all at least heard of bias. Hopefully, we all acknowledge that our decisions are often subject to bias in one way or another.
Noise, on the other hand, can be a bit trickier. While noise and bias can occur at the same time and are loosely connected, they remain two very different phenomena.
Using the diagram below, we can visualize exactly what’s happening in each situation.
Scenario: Four teams are at an archery range. Each team has five people, all sharing the same bow. The targets in the image show the results of each team.
Bias: Systematic deviation
Bias is the unwanted, but predictable, variability in a decision-making system.
If you look at the targets in the image above, Team A and Team B look similar. Team A has hit close to the target pretty consistently. While Team B has missed the target, they all missed with relatively the same variability.
In both cases, you could predict where the next shot would likely be for each team. Team A would probably hit near the target again, while Team B would be slightly off-center. We could also hypothesize about why the variation exists.
Perhaps there’s a flaw in Team B’s bow or maybe they fired their shots from a different position. Either way, we can figure out why Team B is off-target and how their shots ended up where they did.
When we know what the target is, bias is easy to predict and identify.
Noise: Random scatter
Noise is the unexpected and unwanted variability in a decision-making system.
While bias is predictable, noise is not. We can identify noise even without knowing what the target is, though.
Team C’s results are an example of noise. The shots are widely scattered with no apparent pattern or consistency. Based on these results, it would be nearly impossible to accurately predict where a sixth shot might end up.
In addition, we have no idea why Team C missed the target like this. Obviously, none of them were able to hit the target but is that because they’re all very bad archers, or is something else affecting their aim? Are they all experiencing the same effect? Are there multiple effects in play? One could be a bad shot, one could have never used a bow before, one could have sneezed when they released the arrow – we don’t know.
Systematically deviant random scatter
So decisions can be accurate, biased, noisy, or biased and noisy. As if there weren’t enough to worry about already, right?
Team D’s target is both biased and noisy. Each of the shots is systematically off-target, but they’re also widely scattered. If Team D were to take another shot, you may be able to offer a general idea of where it would hit, but with less precision than your prediction for Team B because of the added influence of noise.
Noise would also make it more difficult to determine why the variability existed. Team D could have a flawed bow, which accounts for the bias, but that’s not the only factor affecting their aim.
Unwanted vs. desired variabilities
Not all variability is unwanted.
There are many situations where diverse judgments are encouraged and even necessary. When it comes to competition or innovation, variability can be a good thing. If you’re competing with another company’s service or product, you need a different conclusion to come out ahead.
Noise, in terms of this post, really refers to system noise – that is:
“…unwanted variability in judgments that should ideally be identical…”
Noise in action: Be quiet! I’m trying to think.
Imagine you’re interviewing a prospective employee. Let’s call him Jason. You’ll be conducting the interview with three other colleagues (Clark, Bruce, and Diana) who all have similar levels of experience and familiarity with the hiring process.
During the interview, you find Jason to be articulate and composed. You aren’t completely won over, but he’s a decent candidate who would likely do the job well. You’d like to shortlist him, but continue interviewing candidates.
Your three colleagues, however, disagree:
- Clark: Jason is enthusiastic but also inexperienced, and we need someone who can take ownership of the role. He seems like a good fit for the company, but I don’t think he’s ready for the responsibility of this position. We should move on to other candidates, but keep him in mind for future openings.
- Diana: He comes across as arrogant and aloof. He doesn’t respond well to criticism and seems more interested in doing things his own way than collaborating with a team. I’m concerned his impulsiveness will result in reckless, costly decisions down the line. I don’t think we should hire him at all.
- Bruce: He lacks experience, but he has all the necessary skills. With proper mentorship, he could learn on the job and potentially move up to a management position in a few years. His thinking is unconventional, but it’s good for the team to test new ideas. I don’t think we need to interview anyone else. Let’s make him an offer.
You were all in the same interview, talking to the same person, with the same background and experiences. You should have ended up with fairly similar impressions, yet you came away with four different and very distinct opinions of Jason’s suitability for the job.
So what happened?
Let me give you a few more details:
- Diana started her day with an email from a top client, blasting the work one of her subordinates did for them.
- While Clark isn’t consciously aware of it, Jason does bear a passing resemblance to Clark’s younger brother, who usually means well but often gets in over his head.
- Bruce’s hometown baseball team, the Knights, won the playoff game a few days ago and is headed to the World Series.
You can look at these three facts and say, OK, I get where Diana is coming from. She’s having a bad day because someone on her team messed up, so obviously, she’s going to be more critical of a new candidate. But what does it matter if Bruce’s team won or Clark is thinking of his brother?
Well, more than you think.
According to Kahneman, et al, in a study of 1.5 million judicial decisions over the course of thirty years, it was found that judges made more severe decisions on days that followed the local football team’s loss than they did when the team won.
Other variables that impacted judgment and sentencing decisions included the time of day, if it was the defendant’s birthday, and even the temperature outside.
How many times have you found yourself being kind to a stranger who happened to be nearby when you got some good news? Disliked someone you just met because they wore the same perfume as your grandmother or their voice sounded just like your ex?
Every decision you make is influenced by a million tiny details just like those that you probably aren’t even aware of.
OK, Leks. If I don’t know that noise is affecting my team’s decision-making, what can I do about it?
Simple:
Focus on the process: Cut through the noise
In the above image, we’ve removed the target from each of the examples. Without knowing the end goal, it’s difficult to determine which of the teams were affected by bias. The scattered points, however, still clearly indicate the presence of noise, whether we know what the target was or not.
The Noise Audit Workflow in the next section will help you determine the amount of noise in your decision-making process, but you can’t run an audit every time you need to make a decision.
Instead, you can proactively work to reduce the amount of noise in your decision-making processes by focusing on the process.
A good process should be designed to be usable by pretty much anyone with minimal training. There have been instances here at Process Street where members of the content team have had to swap or take over a project from someone else. We’re able to do that easily in large part because we have a defined process and set parameters for our content creation process.
No matter who’s writing the post, the same tasks will be completed every time. There will be collected research, an outline, title ideas, image requests, and so on. If I got arrested tonight for breaking into the cathedral and destroying the bells, another writer could finish up the post in my place.
Check your noise with the Process Street Noise Audit Workflow
In their book, Noise: A Flaw in Human Judgment, Kahneman et al, detail a rough noise audit process in one of the appendices.
All respect to Professor Kahneman – inarguably a preeminent psychologist and economist (I mean, Nobel Prize winner, people) – but he’s not in the business of process optimization, so I thought I’d give him a hand with that.
Click here to get your Noise Audit Workflow template!
This workflow will walk you through every step of conducting a noise audit on your team or department, from creating the audit team to resolving the judgment variability with your team.
Dynamic due dates let you set task due dates based on varying factors in individual workflow runs.
For example, analysis of the study results should take about three months. Within the workflow, the deadline for reporting the study’s findings is automatically set for three months after the task “Compile the questionnaire answers” is completed.
In addition, conditional logic makes it very simple to adapt the workflow to use cases. With this feature, you can hide unnecessary or irrelevant fields so users only see what they need to see.
So, let’s say the HR team does their audit without an external consultant, but the Sales team decides it’d be a good idea. When HR says they won’t be using an external consultant, the fields to enter consultancy details will remain hidden. The Sales team answers that they will use an external consultant, so they are then prompted to fill out fields for the consultant’s name, email, and the agreement contract.
Judgment variability: Key takeaways
Judgment variability in organizations can be a serious issue when you need your teams to provide consistent decisions in similar situations.
Conducting regular noise audits of your teams will keep you aware of what causes these variabilities and enable you to implement processes and procedures to reduce the level of noise in your decision-making frameworks.
How much noise do you think is in your decision-making framework?
The post Eliminate Noise & Make Better Decisions (+ Free Noise Audit Template) first appeared on Process Street | Checklist, Workflow and SOP Software.
0 Commentaires