Forecasters are notoriously overconfident. This applies to nearly everyone who predicts anything, not just stock analysts. A few fields, like meteorology, have gotten a handle on the uncertainty in their forecasts, but this remains the exception rather than the rule.
Having no good quantitative idea of uncertainty, there is an almost universal tendency for people to understate it. Thus, they overestimate the precision of their own knowledge and contribute to decisions that later become subject to unwelcome surprises.
A solution to this problem involves some better understanding of how to treat uncertainties and a realization that our desire for preciseness in such an unpredictable world may be leading us astray.
E.C. Capen illustrated the problem in 1976 with a quiz that asks takers to state 90% confidence intervals for a variety of things – the length of the Golden Gate bridge, the number of cars in California, etc. A winning score is 9 out of 10 right. 10 out of 10 indicates that the taker was underconfident, choosing ranges that are too wide.
Ventana colleague Bill Arthur has been giving the quiz to clients for years. In fact, it turns out that the vast majority of takers are overconfident in their knowledge – they choose ranges that are too narrow, and get only a three or four questions right. CEOs are the worst – if you score zero out of 10, you’re c-suite material.
My kids and I took the test last year. Using what we learned, we expanded the variance on our guesses of the weight of a giant pumpkin at the local coop – and as a result, brought the monster home.
Now that I’ve taken the test a few times, it spoils the fun, so last time I was in a room for the event, I doodled an updated quiz. Here’s your chance to calibrate your confidence intervals:
For each question, specify a range (minimum and maximum value) within which you are 80% certain that the true answer lies. In other words, in an ideal set of responses, 8 out of 10 answers will contain the truth within your range.
The question is, “what was the winning time in the first Tour de France bicycle race, in 1903?”
Your answer is, “between 1 hour and 1 day.”
Your answer is wrong, because the truth (94 hours, 33 minutes, 14 seconds) does not lie within your range.
Note that it doesn’t help to know a lot about the subject matter – precise knowledge merely requires you to narrow your intervals in order to be correct 80% of the time.
Now the questions:
- What is the wingspan of an Airbus A380-800 superjumbo jet?
- What is the mean distance from the earth to the moon?
- In what year did the Russians launch Sputnik?
- In what year did Alaric lead the Visigoths in the Sack of Rome?
- How many career home runs did baseball giant Babe Ruth hit?
- How many iPhones did Apple sell in FY 2007, its year of introduction?
- How many transistors were on a 1993 Intel Pentium CPU chip?
- How many sheep were in New Zealand on 30 June 2006?
- What is the USGA-regulated minimum diameter of a golf ball?
- How tall is Victoria Falls on the Zambezi River?
Be sure to write down your answers (otherwise it’s too easy to rationalize ex post). No googling!
Answers at the end of next week.
*Update: edited slightly for greater clarity.
8 thoughts on “Calibrate your confidence bounds: an updated Capen Quiz”
If anyone knows a tool that can handle range answers, I’ll convert this to an online quiz.
See also: http://blogs.hbr.org/fox/2012/06/economists-are-overconfident-s.html h/t http://makingsense.facilitatedsystems.com/2012/07/plot-data.html