Noise: A Flaw in Human Judgment
Authors: Daniel Kahneman, Olivier Sibony, Cass R. Sunstein, Daniel Kahneman, Olivier Sibony, Cass R. Sunstein
 (Z-Library)_thumbnail.jpg)
Overview
This book explores the detrimental effects of “noise”—unwanted variability in judgments—in professional and everyday decision-making. We argue that noise is a pervasive and often overlooked source of error, distinct from bias, with significant consequences in various fields, including medicine, law, business, and public policy. We explore the psychology of noise, examining how heuristics, biases, and the limitations of human judgment contribute to its prevalence. We then address the challenge of reducing noise, introducing “decision hygiene” as a set of tools and techniques to improve the quality and consistency of judgments. Case studies in different domains demonstrate the value of applying these strategies. We examine how better judges can be identified, and explore methods of debiasing, the sequencing of information to avoid premature conclusions, the power of aggregating independent judgments, the benefits of guidelines and standards, and the advantages of structuring complex decisions. We also address potential objections to noise reduction, such as cost, fairness, and the importance of flexibility in decision-making, suggesting ways to balance these competing values. Throughout the book, we emphasize the importance of “noise audits” to identify the prevalence of noise within organizations and the potential gains from reducing it. We introduce the mediating assessments protocol (MAP), a general approach to decision-making to improve judgments and prevent error. This work is relevant to the field of artificial intelligence (AI), as algorithms, by their very nature, eliminate noise and offer an alternative to human judgment where variability is a concern. The insights presented here offer principles and techniques that can be implemented not only in traditional organizations but also in designing and using algorithms to improve the quality and consistency of decisions, promoting fairness and reducing errors.
Book Outline
1. Crime and Noisy Punishment
Unwanted variability in judgments of the same case (“noise”) exists across many domains where consistency is expected. It produces unfairness and undermines the rule of law in the criminal justice system, and it can result in big economic losses in the private sector.
Key concept: Wherever there is judgment, there is noise—and more of it than you think.
2. A Noisy System
A “noise audit” can identify the magnitude of system noise in an organization. When considering noise, organizations wrongly assume that another expert would produce a similar judgment when faced with the same problem. However, human judgment is inherently variable, and this variability produces noise.
Key concept: System noise is a problem of systems, which are organizations, not markets.
3. Singular Decisions
Many important decisions are unique or “singular,” which might lead you to think they’re immune to noise. However, noise exists wherever there is judgment, even with singular decisions.
Key concept: A singular decision is a recurrent decision that happens only once.
4. Matters of Judgment
Judgment aims to achieve accuracy, but there are always several possible interpretations of the same evidence, which means that different judges may form different views.
Key concept: Judgment can be described as measurement in which the instrument is a human mind.
5. Measuring Error
Bias and noise both contribute to error, but noise reduction and bias reduction are equally important and have the same impact on overall error. Counterintuitively, noise reduction doesn’t make predictions more accurate, but more precise – a reduction of noise is just as valuable as a reduction of bias.
Key concept: Overall Error (MSE) = Bias² + Noise²
6. The Analysis of Noise
Noise can be decomposed into level noise (variability in average levels of judgment) and pattern noise (variability in how different judges respond to particular cases, relative to their own average).
Key concept: System Noise² = Level Noise² + Pattern Noise²
7. Occasion Noise
Judgments of the same problem by the same person can be different at different times due to occasion noise. This is driven by numerous factors such as mood, stress, or even outside temperature, making all judgments inherently noisy.
Key concept: You are not the same person at all times.
8. How Groups Amplify Noise
While aggregating judgments reduces noise, group dynamics can amplify it. Seemingly irrelevant factors can cause the same group to make vastly different judgments.
Key concept: Responses made by a subject are sampled from an internal probability distribution, rather than deterministically selected on the basis of all the knowledge a subject has.
9. Judgments and Models
Professional judgments are frequently inferior to algorithms or even simple formulas. This is because while human judgment can incorporate complex rules and subtleties, it also incorporates more noise.
Key concept: Where causality is plausible, our mind easily turns a correlation, however low, into a causal and explanatory force.
10. Noiseless Rules
Simple, noise-free models such as equally weighted linear combinations often perform as well as complex formulas like multiple regression. This is because “optimal” formulas adjust themselves to every fluke of the original sample and then perform less well “out of sample.”
Key concept: “we do not need models more precise than our measurements.”
11. Objective Ignorance
Objective ignorance is the limit on how much anyone can know at any given time, which means perfect prediction is impossible, regardless of the forecaster’s skill or of the amount of data.
Key concept: Wherever there is prediction, there is ignorance, and probably more of it than you think.
12. The Valley of the Normal
Objective ignorance makes many events unpredictable. However, because of our tendency to explain events causally, focusing on one explanation after the fact, even events that could not have been predicted can appear explainable and predictable in hindsight.
Key concept: In the valley of the normal, events unfold just like the Joneses’ eviction: they appear normal in hindsight, although they were not expected, and although we could not have predicted them.
13. Heuristics, Biases, and Noise
Heuristics and biases such as substitution, conclusion bias, and excessive coherence contribute to both statistical bias and noise. They create systematic errors in judgment and make judgments overly sensitive to irrelevant information.
Key concept: A heuristic for answering a difficult question is to find the answer to an easier one.
14. The Matching Operation
Matching is the process of finding a value on a scale that best represents our subjective impression. Matching is effective and versatile, but can lead to noise because of our limited ability to translate subjective impressions to specific values on a scale.
Key concept: Judgment [is] an operation that assigns a value on a scale to a subjective impression (or to an aspect of an impression). Matching is an essential part of that operation.
15. Scales
Scales used for judgments can be a major source of noise when the scales themselves are poorly defined. Anchoring on arbitrary values affects judgments on ratio scales (scales with a meaningful zero and no upper bound) like dollar amounts, increasing noise.
Key concept: “Shared Outrage and Erratic Awards: The Psychology of Punitive Damages.”
16. Patterns
Multiple, conflicting pieces of evidence make judgment difficult and increase noise. This is because ambiguity allows different individuals to construct different narratives and arrive at different judgments, leading to larger variability across raters.
Key concept: The rule is simple: if there is more than one way to see anything, people will vary in how they see it.
17. The Sources of Noise
System noise has three components: level noise, stable pattern noise, and occasion noise. Stable pattern noise tends to be larger than the other components. Occasion noise is driven by context, but other sources of noise, related to the individual, are stable and contribute to pattern noise.
Key concept: System Noise² = Level Noise² + (Stable Pattern Noise)² + Occasion Noise²
18. Better Judges for Better Judgments
Better judges tend to have relevant experience, higher intelligence (although this is not the whole story), and actively open-minded thinking, which involves constantly updating beliefs and a desire to be corrected.
Key concept: “They tend to be actively open-minded and willing to learn from new information.”
19. Debiasing and Decision Hygiene
Debiasing interventions are useful in situations where a particular bias is present, but not always. Noise reduction strategies, by contrast, aim to prevent an unspecified range of errors in advance, much like preventive hygiene practices in medicine. Decision observation is a promising strategy for real-time detection of biases.
Key concept: Noise reduction is to debiasing what preventive hygiene measures are to medical treatment: the goal is to prevent an unspecified range of potential errors before they occur.
20. Sequencing Information in Forensic Science
Sequencing information appropriately is a crucial decision hygiene strategy that can be especially effective in forensic science. Because the search for coherence makes us form early impressions, early exposure to irrelevant information might contaminate judgments. It is important to avoid premature conclusions, especially in high-stakes situations.
Key concept: “Sequencing information to limit the formation of premature intuitions”
21. Selection and Aggregation in Forecasting
Effective forecasting is best achieved through statistical thinking, by structuring the prediction problem, and by combining multiple independent forecasts, leveraging the “wisdom of crowds.” Continuous updates to forecasts, known as perpetual beta, allow forecasters to refine predictions as new information becomes available.
Key concept: “try, fail, analyze, adjust, try again.”
22. Guidelines in Medicine
Noise is pervasive in medical judgments. Simple decision guidelines like the Apgar score can reduce noise by clarifying diagnostic criteria, decomposing a complex decision into a number of sub-judgments, and focusing on the relevant variables.
Key concept: The goal of judgment is accuracy, not individual expression.
23. Defining the Scale in Performance Ratings
Level noise is a major factor in performance ratings, and rankings help reduce this noise. Forced ranking can help eliminate noise, but forced distributions create other problems when absolute judgments are mapped to a relative scale.
Key concept: Rankings are less noisy than ratings.
24. Structure in Hiring
Structuring hiring judgments makes them less noisy. A structured process is characterized by decomposition into independent mediating assessments, independent data collection for each mediating assessment, and a final holistic judgment that integrates the judgments made on each assessment.
Key concept: Do not exclude intuition, but delay it.
25. The Mediating Assessments Protocol
The mediating assessments protocol (MAP) helps organizations improve option evaluation. MAP decomposes the evaluation problem into a small number of mediating assessments, collects independent judgments about the assessments, and uses those judgments to inform a final, holistic discussion.
Key concept: Options are like candidates.
26. The Costs of Noise Reduction
There are good reasons to accept some noise, which include the costs of reducing it, the possibility that an effort to reduce noise might increase bias, and the need to treat people with respect and dignity. However, in many cases the benefits of reducing noise far outweigh the cost.
Key concept: Noise may be unwanted, other things being equal. But other things might not be equal…
27. Dignity
Some noise reduction strategies, such as firm rules or algorithms, might prevent an organization from incorporating changing moral values. However, many noise-reduction strategies such as aggregation allow organizations to remain receptive to evolving values.
Key concept: Some noise-reduction strategies would seem unable to make space for [changing values]…
28. Rules or Standards?
Rules eliminate or restrict discretion and hence noise. Standards provide greater freedom for making judgments on a case-by-case basis and for adapting to unique circumstances. But standards create noise, which often becomes intolerable. When that happens, organizations shift toward rules.
Key concept: Rules simplify life and reduce noise.
Essential Questions
1. What is noise, and why is it a problem?
Noise, defined as unwanted variability in judgment, significantly impacts decision quality across various domains. It’s distinct from bias, which represents systematic errors. While bias leads to predictable deviations from the correct answer, noise introduces unpredictable scatter. Both contribute to overall error, and surprisingly, reducing noise can be just as important as reducing bias in improving judgment. Noise arises from multiple sources, including individual differences in judges, variations in the same judge over time (occasion noise), and the way people interpret and apply scales. The pervasiveness of noise has significant implications for organizations and individuals, leading to unfair outcomes, inconsistent practices, and reduced trust in systems.
2. What are the psychological mechanisms that cause noise?
The psychology of noise is rooted in the limitations of human judgment. Heuristics, which are mental shortcuts used to simplify complex problems, often introduce both bias and noise. Conclusion biases lead to prejudgments that color the interpretation of evidence. Our tendency to seek coherence makes us overly reliant on initial impressions (excessive coherence) and slow to revise our beliefs. The matching operation, used to translate subjective impressions onto a scale, introduces noise because of the inherent difficulty in mapping a feeling or impression to a specific value. These psychological mechanisms make us susceptible to noise, even in tasks where we believe we are being objective.
3. What is decision hygiene, and how can it reduce noise?
Decision hygiene is a set of tools and techniques to reduce noise in judgments, analogous to preventive hygiene measures in medicine. Key decision hygiene strategies include structuring judgments by breaking them down into independent assessments, using a shared scale grounded in an outside view (considering a problem statistically as a member of a class of similar problems), sequencing information to avoid premature conclusions, and aggregating independent judgments. These strategies aim to improve the quality and consistency of judgments by limiting the impact of biases, reducing variability, and promoting a more objective approach to decision-making.
4. What are the main objections to noise reduction, and how should they be addressed?
While noise reduction is generally desirable, there are legitimate objections to some noise-reduction efforts. These include the high cost of implementing some strategies, the risk that efforts to reduce noise might introduce other errors (such as bias), the value of treating people with respect and dignity (allowing individualized treatment, even if noisy), and the importance of maintaining flexibility to accommodate evolving moral values. There are also concerns that algorithms, though noise-free, might embed bias or squelch creativity. When considering these objections, it’s important to consider specific noise-reduction strategies and to balance the costs of noise against the costs of reducing it.
5. What are the broader implications of noise reduction for organizations, individuals, and the field of AI?
The implications of noise reduction are wide-ranging. A less noisy world would mean greater fairness in sentencing, hiring, and other evaluative judgments. It would improve the accuracy of medical diagnoses, forecasts, and risk assessments. It would reduce costly errors in business decisions and increase the effectiveness of public policy. For the field of AI, recognizing the pervasiveness of noise reinforces the value of algorithms as a potential tool for improving judgment, but also emphasizes the importance of careful algorithm design and testing to avoid the encoding of biases. The study of noise offers a new lens for understanding and reducing error, with the ultimate goal of making better decisions and improving the quality of judgments across various domains.
1. What is noise, and why is it a problem?
Noise, defined as unwanted variability in judgment, significantly impacts decision quality across various domains. It’s distinct from bias, which represents systematic errors. While bias leads to predictable deviations from the correct answer, noise introduces unpredictable scatter. Both contribute to overall error, and surprisingly, reducing noise can be just as important as reducing bias in improving judgment. Noise arises from multiple sources, including individual differences in judges, variations in the same judge over time (occasion noise), and the way people interpret and apply scales. The pervasiveness of noise has significant implications for organizations and individuals, leading to unfair outcomes, inconsistent practices, and reduced trust in systems.
2. What are the psychological mechanisms that cause noise?
The psychology of noise is rooted in the limitations of human judgment. Heuristics, which are mental shortcuts used to simplify complex problems, often introduce both bias and noise. Conclusion biases lead to prejudgments that color the interpretation of evidence. Our tendency to seek coherence makes us overly reliant on initial impressions (excessive coherence) and slow to revise our beliefs. The matching operation, used to translate subjective impressions onto a scale, introduces noise because of the inherent difficulty in mapping a feeling or impression to a specific value. These psychological mechanisms make us susceptible to noise, even in tasks where we believe we are being objective.
3. What is decision hygiene, and how can it reduce noise?
Decision hygiene is a set of tools and techniques to reduce noise in judgments, analogous to preventive hygiene measures in medicine. Key decision hygiene strategies include structuring judgments by breaking them down into independent assessments, using a shared scale grounded in an outside view (considering a problem statistically as a member of a class of similar problems), sequencing information to avoid premature conclusions, and aggregating independent judgments. These strategies aim to improve the quality and consistency of judgments by limiting the impact of biases, reducing variability, and promoting a more objective approach to decision-making.
4. What are the main objections to noise reduction, and how should they be addressed?
While noise reduction is generally desirable, there are legitimate objections to some noise-reduction efforts. These include the high cost of implementing some strategies, the risk that efforts to reduce noise might introduce other errors (such as bias), the value of treating people with respect and dignity (allowing individualized treatment, even if noisy), and the importance of maintaining flexibility to accommodate evolving moral values. There are also concerns that algorithms, though noise-free, might embed bias or squelch creativity. When considering these objections, it’s important to consider specific noise-reduction strategies and to balance the costs of noise against the costs of reducing it.
5. What are the broader implications of noise reduction for organizations, individuals, and the field of AI?
The implications of noise reduction are wide-ranging. A less noisy world would mean greater fairness in sentencing, hiring, and other evaluative judgments. It would improve the accuracy of medical diagnoses, forecasts, and risk assessments. It would reduce costly errors in business decisions and increase the effectiveness of public policy. For the field of AI, recognizing the pervasiveness of noise reinforces the value of algorithms as a potential tool for improving judgment, but also emphasizes the importance of careful algorithm design and testing to avoid the encoding of biases. The study of noise offers a new lens for understanding and reducing error, with the ultimate goal of making better decisions and improving the quality of judgments across various domains.
Key Takeaways
1. The mediating assessments protocol (MAP) improves the quality of decisions.
MAP improves decision quality by structuring complex problems into manageable components. It allows for independent assessments on each dimension, delaying holistic judgment to prevent the halo effect and other biases from contaminating the process. Using a case scale for assessments ensures a shared frame of reference, making relative rather than absolute judgments. This approach limits the impact of individual biases and makes judgments less noisy. Finally, aggregating independent judgments reduces noise and promotes a better use of information.
Practical Application:
In product design, a team might use MAP to evaluate several design proposals. They would decompose the evaluation into independent criteria such as usability, aesthetics, cost, and manufacturability. Independent teams would assess each proposal on each criterion, using a case scale with reference products if possible. The final decision would integrate these independent assessments.
2. Independent judgments are crucial for effective group decision-making.
Group discussions can significantly amplify noise due to social influences, informational cascades, and group polarization. Early opinions often disproportionately shape the direction of the discussion, even when those opinions are poorly supported by evidence. Initial statements in a meeting can act as anchors or generate cascades, leading the group toward premature and potentially inaccurate conclusions. By eliciting and aggregating independent opinions before the discussion, you can mitigate the harmful effects of social influence noise and make better use of the collective wisdom of the group.
Practical Application:
When leading a meeting, prioritize independent generation of ideas before open discussion. For example, if the team is considering product features, have each member silently list their top three ideas. Then, collect and display all the ideas without attribution, before opening the floor for discussion. This helps to mitigate the influence of cascades and social pressures.
3. Algorithms can be biased, too.
Algorithms eliminate noise and can be very effective in domains with ample data. However, algorithms can also encode biases from the data on which they are trained, and those biases can then be amplified and repeated at scale. Moreover, many algorithms make decisions mechanically without consideration for contextual factors that might be obvious to a human observer (“broken-leg” exceptions). Careful algorithm design, validation, and ongoing monitoring can mitigate the risk that algorithms produce unfair or inappropriate outcomes.
Practical Application:
In AI safety, use decision hygiene principles when designing algorithms for high-stakes applications, such as loan approvals or medical diagnoses. Use multiple, diverse data sets to train algorithms, ensuring data quality and minimizing the risk that biases from any single data set are encoded in the algorithm. Conduct rigorous testing and validation using data not included in the training set.
4. Objective ignorance limits the accuracy of judgments and predictions.
Judgment and prediction are error-prone both because people are noisy and because of objective ignorance: there are facts that we do not and cannot possibly know when we make a decision. Models and algorithms perform better than human judgments by eliminating noise, but they don’t eliminate objective ignorance. The algorithms that predict best in one situation may perform quite poorly in others. As with human judgments, we need to measure the noise of an algorithm, that is, the variability of its performance on the same case on different occasions.
Practical Application:
When evaluating a potential AI solution, consider not only its accuracy on a test set but also its performance in a variety of different settings. Test the AI’s performance with different input data and different use cases, to see how robust it is to unexpected inputs or edge cases. Test the algorithm with different users, to see if it works equally well for different demographic groups.
5. Sequencing information matters.
The sequence in which information is presented can have a large impact on judgment due to cognitive biases such as anchoring, confirmation bias, and excessive coherence. Early exposure to irrelevant information or information of uncertain accuracy or relevance can bias judgments and increase noise. It is important to control the sequence of information, ensuring that important, accurate, and relevant data is given its proper weight in the final judgment.
Practical Application:
When designing a user interface, consider the order in which information is presented to the user. For example, if a form asks for a user’s date of birth, consider displaying aggregate statistics about users’ ages before asking for the individual’s date of birth. This might reduce anchoring effects and ensure greater accuracy. Ensure visual salience is not influencing judgment more than it should.
1. The mediating assessments protocol (MAP) improves the quality of decisions.
MAP improves decision quality by structuring complex problems into manageable components. It allows for independent assessments on each dimension, delaying holistic judgment to prevent the halo effect and other biases from contaminating the process. Using a case scale for assessments ensures a shared frame of reference, making relative rather than absolute judgments. This approach limits the impact of individual biases and makes judgments less noisy. Finally, aggregating independent judgments reduces noise and promotes a better use of information.
Practical Application:
In product design, a team might use MAP to evaluate several design proposals. They would decompose the evaluation into independent criteria such as usability, aesthetics, cost, and manufacturability. Independent teams would assess each proposal on each criterion, using a case scale with reference products if possible. The final decision would integrate these independent assessments.
2. Independent judgments are crucial for effective group decision-making.
Group discussions can significantly amplify noise due to social influences, informational cascades, and group polarization. Early opinions often disproportionately shape the direction of the discussion, even when those opinions are poorly supported by evidence. Initial statements in a meeting can act as anchors or generate cascades, leading the group toward premature and potentially inaccurate conclusions. By eliciting and aggregating independent opinions before the discussion, you can mitigate the harmful effects of social influence noise and make better use of the collective wisdom of the group.
Practical Application:
When leading a meeting, prioritize independent generation of ideas before open discussion. For example, if the team is considering product features, have each member silently list their top three ideas. Then, collect and display all the ideas without attribution, before opening the floor for discussion. This helps to mitigate the influence of cascades and social pressures.
3. Algorithms can be biased, too.
Algorithms eliminate noise and can be very effective in domains with ample data. However, algorithms can also encode biases from the data on which they are trained, and those biases can then be amplified and repeated at scale. Moreover, many algorithms make decisions mechanically without consideration for contextual factors that might be obvious to a human observer (“broken-leg” exceptions). Careful algorithm design, validation, and ongoing monitoring can mitigate the risk that algorithms produce unfair or inappropriate outcomes.
Practical Application:
In AI safety, use decision hygiene principles when designing algorithms for high-stakes applications, such as loan approvals or medical diagnoses. Use multiple, diverse data sets to train algorithms, ensuring data quality and minimizing the risk that biases from any single data set are encoded in the algorithm. Conduct rigorous testing and validation using data not included in the training set.
4. Objective ignorance limits the accuracy of judgments and predictions.
Judgment and prediction are error-prone both because people are noisy and because of objective ignorance: there are facts that we do not and cannot possibly know when we make a decision. Models and algorithms perform better than human judgments by eliminating noise, but they don’t eliminate objective ignorance. The algorithms that predict best in one situation may perform quite poorly in others. As with human judgments, we need to measure the noise of an algorithm, that is, the variability of its performance on the same case on different occasions.
Practical Application:
When evaluating a potential AI solution, consider not only its accuracy on a test set but also its performance in a variety of different settings. Test the AI’s performance with different input data and different use cases, to see how robust it is to unexpected inputs or edge cases. Test the algorithm with different users, to see if it works equally well for different demographic groups.
5. Sequencing information matters.
The sequence in which information is presented can have a large impact on judgment due to cognitive biases such as anchoring, confirmation bias, and excessive coherence. Early exposure to irrelevant information or information of uncertain accuracy or relevance can bias judgments and increase noise. It is important to control the sequence of information, ensuring that important, accurate, and relevant data is given its proper weight in the final judgment.
Practical Application:
When designing a user interface, consider the order in which information is presented to the user. For example, if a form asks for a user’s date of birth, consider displaying aggregate statistics about users’ ages before asking for the individual’s date of birth. This might reduce anchoring effects and ensure greater accuracy. Ensure visual salience is not influencing judgment more than it should.
Memorable Quotes
Introduction. 12
In real-world decisions, the amount of noise is often scandalously high.
Introduction. 16
Wherever there is judgment, there is noise—and more of it than you think.
Chapter 5. 61
Bias and noise play identical roles in the error equation.
Chapter 6. 75
In a perfect world, defendants would face justice; in our world, they face a noisy system.
Chapter 10. 116
“we do not need models more precise than our measurements.”
Introduction. 12
In real-world decisions, the amount of noise is often scandalously high.
Introduction. 16
Wherever there is judgment, there is noise—and more of it than you think.
Chapter 5. 61
Bias and noise play identical roles in the error equation.
Chapter 6. 75
In a perfect world, defendants would face justice; in our world, they face a noisy system.
Chapter 10. 116
“we do not need models more precise than our measurements.”
Comparative Analysis
“Noise” distinguishes itself by focusing on a pervasive but neglected aspect of judgment: its variability. While other books on decision-making, such as “Thinking, Fast and Slow” or “Predictably Irrational,” primarily explore systematic biases, “Noise” emphasizes the random scatter in judgments that undermines accuracy and fairness. It shares some common ground with works on forecasting, like “Superforecasting,” in recognizing the limits of human judgment and advocating for structured approaches and aggregation. However, “Noise” goes further by offering a comprehensive framework for understanding and mitigating noise across various domains, including medicine, law, and business. It also diverges from some popular management literature that celebrates intuition and “gut feelings,” arguing that while intuition has its place, it should be informed, disciplined, and delayed, not relied upon as the primary driver of decisions. The book’s focus on system noise and decision hygiene provides a new lens for evaluating and improving judgments, with direct implications for the design and use of AI algorithms, particularly in contexts where consistency and fairness are paramount.
Reflection
“Noise” provides a valuable framework for understanding and addressing a neglected aspect of judgment. The book’s emphasis on the pervasiveness and cost of noise is compelling, and its proposed solutions, grounded in behavioral science, are practical and actionable. However, the book might overstate the case for algorithms. While algorithms eliminate noise, concerns about their potential for bias are not adequately addressed, especially in contexts involving social or ethical considerations. Moreover, the book’s advocacy for structured processes, while beneficial in many cases, might not always be feasible or desirable, particularly in situations requiring flexibility, creativity, or rapid decision-making. The book’s focus on quantitative, verifiable judgments might underemphasize the role of subjective judgments where a “true value” is less easily defined. Nonetheless, “Noise” makes a significant contribution to the field of decision-making by highlighting the often-hidden problem of noise and offering a powerful set of tools to improve judgment quality and reduce error. It challenges our assumptions about the objectivity and reliability of judgments, prompting a re-evaluation of how we make decisions in both professional and everyday life.
Flashcards
What is noise?
Unwanted variability in judgments; distinct from bias (systematic error).
What is bias?
Systematic error in judgments; distinct from noise (random error).
What is the error equation?
Overall Error (MSE) = Bias² + Noise²
What is a noise audit?
A process for measuring noise in judgments by having multiple judges evaluate the same cases.
What is level noise?
Variability in average judgments by different judges.
What is pattern noise?
Variability in judges’ responses to particular cases.
What is occasion noise?
Variability in judgments of the same case by the same person on different occasions.
What is decision hygiene?
A set of techniques to reduce noise in judgments, analogous to preventive hygiene measures in medicine.
What is structuring in the context of decision hygiene?
Breaking down a complex judgment into smaller, independent sub-judgments.
What is the outside view?
Considering a case statistically as a member of a class of similar cases.
What is noise?
Unwanted variability in judgments; distinct from bias (systematic error).
What is bias?
Systematic error in judgments; distinct from noise (random error).
What is the error equation?
Overall Error (MSE) = Bias² + Noise²
What is a noise audit?
A process for measuring noise in judgments by having multiple judges evaluate the same cases.
What is level noise?
Variability in average judgments by different judges.
What is pattern noise?
Variability in judges’ responses to particular cases.
What is occasion noise?
Variability in judgments of the same case by the same person on different occasions.
What is decision hygiene?
A set of techniques to reduce noise in judgments, analogous to preventive hygiene measures in medicine.
What is structuring in the context of decision hygiene?
Breaking down a complex judgment into smaller, independent sub-judgments.
What is the outside view?
Considering a case statistically as a member of a class of similar cases.