monty hall

I was talking to Christopher Lin at the University of Washington about Mechanical Turk experiments. One issue that came up is the problem of knowing the correct answer from a crowd when most people in the crowd are wrong. For instance, when answering a tricky brain teaser, one might imagine that most people are tricked, so the most popular answer is not correct. Yu An Sun has done some work on this, suggesting the tool of asking for answers in one pass, and then asking people to select from those answers in another pass. The idea is that people may not be tricked as easily if they have options to chose from, since seeing the correct answer may in some way unveil the trick.

I mentioned a similar problem with crowd sourcing which is trying to know if people are lying when it comes to subjective questions, where there is no ground truth at all. For instance, imagine asking people to say which logo design they like better. People may have legitimate differences of opinion. One tool I've heard of for combatting this is bayesian truth serum, which I've mentioned in some other blog post, but the basic idea is to ask people two questions: "what is your opinion?" and "what do you think most other people's opinion will be?".

Christopher then suggested the idea of applying this to the brain teaser problem: asking people for their answer on a brain teaser, and also asking them how they think other people will answer it. The idea being that maybe you can identify the "correct minority" if they're answering with some other answer, but correctly predicting the way in which most people will get it wrong, i.e., showing that they are aware of the trick that most people will fall prey to.

So, I asked 100 turkers to answer the Monty Hall problem, which is tricky, and to predict how other people would answer it.

I phrased it like this:

Imagine a game show. Everyday day, a person gets a chancee to win a prize. The prize is inside one of three boxes. The player always chooses a box, and then the game show host always reveals one of the remaining boxes to be empty, and allows the player to switch their choice to the remaining box if they want.

Should they switch their choice?

• Yes, Switch
• No, Keep First Guess
• It Doesn't Matter

How will other people answer?

• Yes, Switch
• No, Keep First Guess
• It Doesn't Matter

The raw results are here. Note that there are only 91 responses. I only waited 24 hours, and that's how many responses I got in that time. The column with the numbers represents work time in seconds. The "maybe" option refers to "It Doesn't Matter".

Most people answered it wrong. The correct answer is "Yes, Switch". However, the people who answer "Yes, Switch" typically thought that other people would also answer "Yes, Switch", so that kindof goes against my theory.

It's possible that the people who answered "Yes, Switch" were just choosing the first option from both sets of radio buttons, but some of these people spent over a minute thinking about it, so I'm not sure.

I've re-asked the question, this time with no radio buttons, forcing people to actually type something in. I'm hoping that this will make it a little more clear who is cheating (if anyone is cheating).

UPDATE:
I ran it again with this question:

(100 turkers, $0.05/assignment)


Imagine there is a prize in one of three boxes. You chose one of the boxes, and then someone opens one of the other two boxes, revealing nothing inside. You then switch your guess to the other unopened box.

What percent chance do you have of this being the box with the prize?

[                                 ] %

What answer do you think most other people will give?


[                                 ] %

The raw results are here.

The answers look pretty scattered, but most people (58 out of 100) put 50% for the first question, so we might consider only keeping people's answers who correctly guessed 50% for the second question. Doing this we get:

46 said 50%

3 said 66%

2 said 60%

1 said 66.66%

1 said 65%

1 said 33%

1 said 40%

1 said 25%

1 said 75%

The most common answer after 50% is now 66%, which is correct. I didn't tell people how accurate to be, so I suspect the people who said 60% and 65% also meant 66%.

In any case, this seems like a promising way to track down the correct answer from a crowd, even when most people get it wrong.