Quiz from Economist "World in Figures 2006"
2 Sept. 07
CHOOSE MINIMUM & MAXIMUM VALUES SO YOU'RE 50% SURE THAT THE TRUE VALUE IS BETWEEN THEM.
How it works: The questions are taken from "The Economist Pocket World in Figures 2006". The questions are randomly chosen. Please choose upper & lower bounds that would make you 50% sure that the real answer is between those two numbers. So the aim is to get 5/10 of the answers being within the bounds you give.
NOTE: it doesn't matter if you have no idea what the answer is to the questions. It just matters how well you know your own ignorance. In theory, if you get less than half right then you're overconfident, and if you get more than half right then you're underconfident.
Now incentive compatible: Asking for you to get 5/10 answers within your bounds is vulnerable to manipulation. There's a more robust scoring rule: say your lower bound is 0, your upper bound is x, and the true answer is z, then you get a penalty proportionate to the distance between z and x. With this scoring role it's optimal to choose an x so that P(z≤x)=½, i.e. to choose 50% bounds on your beliefs. (By the same logic that the median is the estimator that minimises absolute deviation; & note this assumes you're trying to minimise expected value of the penalty).
(P.S.: don't cheat).
- The question's not well-defined, what do I do? Treat the question as this: "if I were to borrow Tom's book, and look up this number, what would I expect to find?" That's a well-defined question.
- Does it matter that the questions are randomly selected? Yes. If I had intentionally chosen surprising facts from the book, without telling you, then you probably would get less than 50% within your range. That is one of the problems with the FT quiz: you don't know the quiz-setter's method of choosing questions.
- Why 50% confidence intervals? That gives the test of the most statistical power. If you're asked for 90% confidence (as in the FT test), getting 10/10 doesn't really show that you're under-confident, it could be just chance. (Though ... you lose statistical power in proving that you're overconfident).
- What if I know the answer? The answers I have from the book are all to 2 or 3 significant digits. If you think you already know the answer to that precision, then you shouldn't use your answer in calculating your final score, because there's no way you can give 50% confidence bounds.
- Isn't there a degree of freedom in the bounds I choose? Yes, seems there is. If you had a well-defined prior distribution, you could - for example - always set the lower bound at 0 and choose the upper bound u so P(0<=x<=u)=0.5. Yet it seems natural to choose bounds which are somehow symmetric.
- Can you make this incentive compatible? Done, see above.
- See this paper : http://www.sciencedirect.com/science/article/pii/S0167268105001654
- Who do I complain to? t(.)e(.)cunningham(at)lse(.)ac(.)uk
Answers (don't look).