Mental Pivot #54: Calibrating Confidence
Assessing if you know what you know, notes on starting an attention diet, and Kevin Kelly’s taxonomy for technology problems.
If you’d like to improve the quality of your decision-making, Annie Duke’s 2018 book, “Thinking in Bets,” isn’t a bad place to find inspiration. Duke, a former professional poker player, knows a thing or two about uncovering the best choice when confronted with uncertainty.
While revisiting my notes on her book, I was struck by her thoughts on confidence. She writes:
We would be better served as communicators and decision-makers if we thought less about whether we are confident in our beliefs and more about how confident we are. Instead of thinking of confidence as all-or-nothing (“I’m confident” or “I’m not confident”), our expression of our confidence would then capture all the shades of grey in between.
One way to “capture all the shades of grey” is by calibrating one’s certainty in a belief with a supplemental confidence score. Duke describes this tactic in detail:
What if, in addition to expressing what we believe, we also rated our level of confidence about the accuracy of our belief on a scale of zero to ten? Zero would mean we are certain a belief is not true. Ten would mean we are certain that our belief is true. A zero-to-ten scale translates directly to percentages. If you think the belief rates a three, that means you are 30% sure the belief is accurate. A nine means you are 90% sure. So instead of saying to ourselves, “Citizen Kane won the Oscar for best picture,” we would say, “I think Citizen Kane won the Oscar for best picture, but I’m only a six on that.” Or “I’m 60% that Citizen Kane won the Oscar for best picture.” That means your level of certainty is such that 40% of the time it will turn out that Citizen Kane did not win the best-picture Oscar. Forcing ourselves to express how sure we are of our beliefs brings to plain sight the probabilistic nature of those beliefs, that what we believe is almost never 100% or 0% accurate but, rather, somewhere in between.
This approach serves a dual-purpose of tempering epistemic overconfidence and keeping us open to other possibilities.
Of course, attaching a confidence-level to one’s beliefs is antithetical to much that we’ve been taught. Superficially, the world craves and rewards certainty and conviction. I don’t recall any school teacher ever asking me on an exam to qualify an answer with a confidence score (though it would have been a welcome addition to many test formats!).
Still, one has to wonder about the effect of this approach on interpersonal discourse and decision-making were we all to adopt this behavior. At the very least, it would inject more humility into the equation, which might be what the contemporary zeitgeist needs.
A few years back, Andrew and Michael Maboussin created an online tool to assess confidence and belief. You can try it out firsthand by participating in their confidence calibration exercise (no registration required, exercise takes a few minutes).
Participants answer 50 true/false questions and indicate a confidence score for each answered question (i.e., “I am 70% certain of my answer). Upon completion, participants receive a report detailing their performance along with a mean confidence score (i.e., what you thought you knew) vs. your mean accuracy (i.e., what you actually got right).
Ideally, your confidence and accuracy will match up, which means that you, in fact, know what you know (well-calibrated confidence!). If your confidence score is significantly higher than your accuracy, that might suggest that you are overconfident (you expected to perform better than you actually did). If your confidence score is significantly lower than your accuracy, that might signal a lack of confidence. And if you max out with a dual 100% score for confidence and accuracy, well, maybe I should be reading YOUR newsletter, not the other way around. For what it’s worth, the Maboussin tool (along with other studies) have found a mean 10% overconfidence score. This is based on data from thousands of participants.
I ran the exercise and received a mean confidence of 62% with an answer accuracy of 68%. My (paltry) 14 high-confidence score answers, defined as 90-100% confidence, were all answered correctly. Do I have a healthy appreciation for the limits of my knowledge (or was I biased because I think about these topics regularly)? I’m going to hedge and say “maybe” with a 50% confidence-score to that question.
Further reading:
The Overconfidence Effect: Wikipedia entry on the cognitive bias that some believe to be the most destructive and pernicious of human failings.
Racehorses and Psychopaths: Tom Morgan’s recent piece on a famous study of racehorse handicappers, overconfidence bias, and Julia Galef’s scout mindset. This article led me to the confidence calibration exercise linked above.
Why Are We So Confident?: Andrew Mauboussin writes about the challenges of studying overconfidence.
Now onto this week's recommendations…
Thinking Tools:
How to Start an Attention Diet?: Rishikesh Sreehari writes about the importance of deliberate media consumption and offers ideas on how to start one. The author also writes the excellent 10+1 Things newsletter (see my spotlight on his newsletter below in the Odds & Ends section).
How to Successfully Scope and Finish (Personal) Projects: Jason Tu’s article is geared towards the creation of indie games, but the lessons he presents are applicable to any hobby or side project: develop a plan, run weekly sprints to make continuous progress (he uses the scrum framework), and cultivate your intrinsic motivation and joy.
Podcasts Are My New Wikipedia: Wenbing Fang, creator of podcast search engine Listen Notes, discusses how podcasts are a key part of his “informal learning” process (informal learning in this context means getting a basic introduction or overview—this type of learning can be supplemented later with more serious materials). Of particular interest: Fang doesn’t subscribe to individual podcasts, but rather maintains a “master feed” into which he curates individual episodes. I like this approach and plan to try it out.
Reading Enrichment:
Class 1 and Class 2 Problems: Kevin Kelly (founder of Wired) ponders two types of technology problems: the first are problems where technology does not work properly, the second are problems where technology works too well. With numerous examples drawn from technologies like facial recognition and autonomous vehicles, Kelly provides readers with a framework for anticipating and responding to technological innovation.
The Code that Controls Your Money: Millions of lines of COBOLcode undergird our financial system, but many of those who designed those systems are long-retired. What happens to these critical legacy systems given the shortage of expertise in a venerable language and the high costs of rewriting and upgrading these systems?
Facebook: What to Do: Scott Galloway on social media and a U.S. regulation from 1996 known as Section 230: “Section 230 creates an imbalance between protection and liability…How do we redraw Section 230’s outdated boundary in a way that protects society from the harms of social media companies while maintaining their economic vitality?”
How Four Oxford Women Transformed Philosophy: In the 1940s, a quartet of remarkable women challenged the Oxford orthodoxy in moral philosophy that values are not real and sought a serious appraisal of ethics and moral standards. This article is a review of Benjamin Lipscomb’s new group biography, “The Women Are Up to Something” and sheds light on four important thinkers that I haven’t heard enough about.
What Collective Narcissism Does to Society: Psychologist Scott Barry Kaufman spotlights the work of Agnieszka Golec de Zavala and her efforts to understand group narcissism (defined as “a belief that the exaggerated greatness of one’s group is not sufficiently recognized by others.”). Once seen as a fringe problem, Golec de Zavala is concerned about the ubiquity of group narcissism and its negative side effects (prejudice, radicalism, and even violence).
Odds & Ends:
10+1 Things is a weekly curated newsletter by engineer and autodidact, Rishikesh Sreehari. Favored topics include technology, business, sustainability, personal development and art (in other words, it’s perfect for Mental Pivot readers). This week’s issue looks at mimicking canine disease detection with cutting-edge technology, the fascinating story behind the humble graphite pencil, and policy recommendations for counteracting the environmental costs of cryptocurrency power consumption.
The Gallery of Regrettable Food is a delightful time-capsule that takes visitors back to 1940s, 1950s, and 1960s America and the questionable cookbooks and food trends of the time (unholy casseroles? pineapple rings? aspics?). The site collects scans of period recipe books and adds snarky commentary to the mix. It’s a personal favorite and one of my longest tenured browser bookmarks. The food gallery is part of a larger project chronicling mid-century Americana known as “The Institute of Official Cheer.”—worth a visit if you love the aesthetics of the period.
Cross-Promotions:
The Sample: A newsletter discovery tool. Based on your interests and feedback, The Sample sends a new newsletter recommendation to your inbox on a daily or weekly basis.
The Veggie Digest: A weekly newsletter about the latest trends in sustainable food innovation. My daughter, an environmental policy student, writes it and I periodically contribute to it.
Thank you for subscribing to the Mental Pivot Newsletter. If you’re enjoying it, be sure to share it with your friends and spread the word.
I want to be able to deliver a top-notch newsletter to all of you. To that end, I’d love to hear your thoughts on what’s working, what doesn’t, and things you’d like to see more of. You can reach me by replying directly to this email or by adding a comment on Substack.
If this newsletter was forwarded to you, visit this link to subscribe.