Summary: Answer the question to the right, and see how other turkers and blog readers answered similar questions. After you answer the question, the red dots will show where people answered "yes".
This is just for fun. Stuff like this has been done before. For instance, this CrowdFlower post shows what names people give to different colors.
I had originally wanted to ask if the dot was "above" the bar, but I feared the result might look to boring, like a rectangle.
Technical tid-bits:
The url in the iframe is exactly the same url I posted on MTurk. The JavaScript detects if it is being viewed on MTurk and does two things. First, if it is being viewed in preview mode, it overlays a grey rectangle with the word "preview" to prevent people from accidentally answering the question before they accept the HIT. Second, it makes it so that answering the question submits the HIT, rather than showing the results. The utility function for submitting the HIT creates a form with the necessary elements, including a dummy result so that MTurk doesn't feel left-out -- this is necessary, since MTurk considers it an error to submit no results. Of course, the actual result is submitted to my server via ajax before submitting the HIT.
I only use one API call to MTurk, and that is to create the HIT. I don't get results, and I don't approve people. I don't get results because they have already been submitted to my server, and I don't approve people because they will automatically be approved after one hour -- this is a parameter when creating the HIT. But Greg, doesn't that mean people can cheat? Well, yes, but I find that people typically don't cheat on questions like this, because they're simple and kindof fun. Also, I wouldn't know if they did cheat. Each person is only allowed to answer the question once, so I can't even see if their results are statistically improbable.
Another tidbit is that I sometimes have unrealistic scalability fears, like "what if lots of people read this post and answer the question, endlessly increasing the size of the data structure until it takes forever for people to download?" So, I made this data structure have a maximum size: there are a finite number of points it will ask about (you can see them arranged on a grid after you answer the question), and it just records the number of yesses and noes for each location. But Greg, that doesn't really limit the size of the data structure, since the number of yesses or noes can grow infinitely. True, but first, that is a scalability issue I'm willing to sleep at night with, and second, JavaScript numbers cannot actually be infinitely large, like Python numbers can. I think JavaScript uses 64-bit floating point numbers under the hood. So at some point, the data structure would just be wrong, rather than too big...
Showing posts with label research. Show all posts
Showing posts with label research. Show all posts
10/13/12
7/6/12
monty hall again
I ran an experiment on mturk here, and then I decided that I made a mistake here. I reran the experiment, and here are the results from that (note that only 90 people did it before the HIT expired).
The results were similar, but not quite as good. The most common answer was again 50%, followed by three-way tie between 100%, 75% and 70%. If we restrict ourselves to just people who guessed that the most common answer would be 50%, then the most common answer is again 50%, followed by a three-way tie between 67%, 33% and 20%. This result isn't as nice as before, where the next most common answer was just 66%. However, both 67% and 33% are more reasonable than 100% or 75%. I'm not sure where 20% is coming from.
I also ran an experiment with the same idea, but a different brain teaser:
Imagine you have 9 identical looking balls, and a weighing scale. All the balls weigh the same except one, which is heavier.
How many weighings on the scale do you need in order to identify the heavier ball?
[ ]
What answer do you think most other people will give?
[ ]
The results from this are here. Note that only 64 people did the HIT before it expired. Also note that there was more variance in how people answered: some people added words, and some of the answers appear to be percentages rather than numbers. The most common answers were 3 and 4 with 19% and 17% respectively. If we restrict ourselves to people who guessed that the most common answer would be 3 or 4, then all the answer are 3 and 4, except one person who answered 8. Only 3 people gave the correct answer of 2, but two of these people guessed that most people would answer 2, and one of these people guessed that most people would answer 1 -- which seems sketchy to me.
So, I'm less confident in this technique. I still think there may be some clever way to get turkers to reason through difficult brain teasers -- maybe it would help to ask people why they gave the answer they did, and have other people rate those bits of reasoning?
The results were similar, but not quite as good. The most common answer was again 50%, followed by three-way tie between 100%, 75% and 70%. If we restrict ourselves to just people who guessed that the most common answer would be 50%, then the most common answer is again 50%, followed by a three-way tie between 67%, 33% and 20%. This result isn't as nice as before, where the next most common answer was just 66%. However, both 67% and 33% are more reasonable than 100% or 75%. I'm not sure where 20% is coming from.
I also ran an experiment with the same idea, but a different brain teaser:
Imagine you have 9 identical looking balls, and a weighing scale. All the balls weigh the same except one, which is heavier.
How many weighings on the scale do you need in order to identify the heavier ball?
[ ]
What answer do you think most other people will give?
[ ]
The results from this are here. Note that only 64 people did the HIT before it expired. Also note that there was more variance in how people answered: some people added words, and some of the answers appear to be percentages rather than numbers. The most common answers were 3 and 4 with 19% and 17% respectively. If we restrict ourselves to people who guessed that the most common answer would be 3 or 4, then all the answer are 3 and 4, except one person who answered 8. Only 3 people gave the correct answer of 2, but two of these people guessed that most people would answer 2, and one of these people guessed that most people would answer 1 -- which seems sketchy to me.
So, I'm less confident in this technique. I still think there may be some clever way to get turkers to reason through difficult brain teasers -- maybe it would help to ask people why they gave the answer they did, and have other people rate those bits of reasoning?
6/21/12
interview game
This is another past experiment. The idea was to have people interview each other, and assess each other's English skills. You would wait for someone to chat with, then you'd chat with them for 10 minutes, and then you would rate their English ability using the smiley slider.
This interface was used for a 1 hour pilot experiment where 3 oDesk people, including myself, interviewed 10 or so oDesk contractors, and they also interviewed each other.
I think the 10 minute time was too long. Apart from that, I'm not sure why this experiment faded away, and nothing more was done with it. I think my energies went into Talent Court shortly after this.
heniac
While going through my old Google App Engine apps and killing them, I stumbled across an old project, and thought I'd take a screenshot to remember it by. The project was called HENIAC (Human - ENIAC). The idea was that there would be a pool of assistants standing by at all hours of the day, and you could create a new chat session, and then "summon" one of them by clicking the "summon to current session" button.
To incentivize having a worker online at all times, workers were paid 20% of their normal hourly wage just to be available. Or rather, when you're available, you would be in a queue, and the person at the head of this queue would be making 20% of their wage, while the next person in line would only be making 10%, then 5%, and so on.. so if the queue already has a few people in it, you might not bother to join.
You could also reserve time at the bottom. If you reserved an hour, then you would automatically be bumped up to the top of the queue during that hour, so you could reserve time and know that you would be paid at least 20% of your wage during that time, even if there was no work.
So what happened with HENIAC? Well, the main problem was that people didn't use it. Here are some speculations about why:
- Our pool of people were not expert enough for the sorts of tasks people would want to hire them for.
- Many "personal admin" type tasks were hampered by not wanting to give passwords and such to this pool.
- People felt uncomfortable telling these people what to do -- one person mentioned that he always felt like he was apologizing before asking for something to be done because he knew that the thing he wanted done was very boring (which is why he was asking someone to do it in the first place).
- Related to this, this same person felt a bit uncomfortable with the social interactions that seemed necessary before and after asking for something to be done, e.g., he felt like he should say "hi, how are you?" rather than start straight in with "do X".
- Because people were not using the system very often, our pool of people were not always there. They would often mark themselves as available, but then essentially leave their computer, assuming nobody would ask them for work, so when people would ask for work, the system would ask each person on the queue, and nobody would respond. This wouldn't happen all the time, but I've seen it happen, and I think the fact that it could happen decreased confidence in availability, leading in a vicious cycle to less use.
6/11/12
monty hall blunder
I asked turkers to solve the Monty Hall problem. I phrased it like this:
If they might have revealed a box with the prize, then the probability breaks down this way: there's a 1/3 chance that I chose the right box, and they reveal an empty box; there's a 1/3 chance that I chose the wrong box, and they reveal an empty box; and there's a 1/3 chance that I chose the wrong box, and they reveal the prize (and don't let me change my guess). Hence, if they reveal an empty box, then there is a 50% chance that the prize is in each of the remaining boxes.
If they never reveal the box with the prize, then the probability breaks down this way: there's a 1/3 chance that I chose the right box, and they reveal an empty box; and there's a 2/3 chance that I chose the wrong box, and they reveal an empty box, meaning that the prize is in the remaining box.
So I've asked turkers again, which this phrasing:
I'm also curious, of the people who have heard it before, how many would believe that it matters whether the host was definitely going to show an empty box, since if the host does show an empty box, it seems equivalent to a host that was definitely going to show an empty box -- especially since we're just asking about a particular instance of the game.
Imagine there is a prize in one of three boxes. You chose one of the boxes, and then someone opens one of the other two boxes, revealing nothing inside. You then switch your guess to the other unopened box. What percent chance do you have of this being the box with the prize?The problem is that, in this scenario, it is unknown whether the person revealing the empty box had any chance of revealing a box with a prize, since presumably if they had opened a box with the prize, they wouldn't have let me switch my guess to that box.
If they might have revealed a box with the prize, then the probability breaks down this way: there's a 1/3 chance that I chose the right box, and they reveal an empty box; there's a 1/3 chance that I chose the wrong box, and they reveal an empty box; and there's a 1/3 chance that I chose the wrong box, and they reveal the prize (and don't let me change my guess). Hence, if they reveal an empty box, then there is a 50% chance that the prize is in each of the remaining boxes.
If they never reveal the box with the prize, then the probability breaks down this way: there's a 1/3 chance that I chose the right box, and they reveal an empty box; and there's a 2/3 chance that I chose the wrong box, and they reveal an empty box, meaning that the prize is in the remaining box.
So I've asked turkers again, which this phrasing:
Imagine a game with a prize in one of three boxes. The game works like this: you choose a box, the host reveals an empty box that you didn't choose, and then you have a chance to switch your guess to the remaining box. What percent chance do you have of getting the prize if you switch to the remaining box?I'm not totally satisfied with this phrasing either. Though, I suspect the outcome on turk will be the same, since I think people typically only get this problem right if they've heard it before, and if they've heard it before, I think they'll assume it's the correct version of the problem, without reading it carefully.
I'm also curious, of the people who have heard it before, how many would believe that it matters whether the host was definitely going to show an empty box, since if the host does show an empty box, it seems equivalent to a host that was definitely going to show an empty box -- especially since we're just asking about a particular instance of the game.
Monty Hall on MTurk
[UPDATE: My phrasing of the problem to turkers is poor. I believe it is important to the problem that the player knows that they'll always be shown an empty box that they didn't choose after making their first guess. If the host opens one of the two remaining boxes randomly, possibly revealing the prize, then I think the odds are in fact 50% of getting the prize by switching your guess. My phrasing does not make this clear. Alas.]
I asked 100 turkers to answer the Monty Hall problem. Most of them got it wrong. However, similar to the baysean truth serum post, I also asked turkers to say how they thought other people would answer it, and this showed some promise of helping identify the correct answer.
The exact question was this:
Imagine there is a prize in one of three boxes. You chose one of the boxes, and then someone opens one of the other two boxes, revealing nothing inside. You then switch your guess to the other unopened box.
What percent chance do you have of this being the box with the prize? [ ]%
What answer do you think most other people will give? [ ]%58 people answered 50%, whereas the correct answer is two thirds. The next most common answer was 100%. (You can see the raw results here.)
However, if we only look at people who correctly predicted how most other people would answer, i.e., people who answered 50% for the second question, then we get 48 people saying 50%, and the next most common answer is 66%. Three people provided this answer, and 4 more people provided answers like 66.66%, 60% and 65%.
Hence, it seems like the baysean truth serum idea may hold promise for extracting the correct answer from a crowd for tricky questions where most people get it wrong. At least, it may give us good reason to investigate a non-majority answer.
Background:
This idea came up while talking with Christopher Lin at the University of Washington about Mechanical Turk experiments. He was interested in the problem of getting a correct answer from a crowd when the majority of people provide the wrong answer, e.g., when a problem is tricky, and most people fall for the trick. Yu-An Sun has done some work on this, suggesting the idea of asking for answers in one pass, and then asking people to select from those answers in another pass. The idea is that people may not be tricked as easily if they have options to chose from, since seeing the correct answer may in some way unveil the trick.
I mentioned a similar problem: how do you know if people are lying when it comes to subjective questions, where there is no ground truth at all? I said that the only traction I've encountered for that problem is the bayesian truth serum trick of asking two questions: "what is your opinion?" and "what do you think most other people's opinion will be?".
Christopher then suggested applying this to the brain teaser problem: asking people for their answer on a brain teaser, and also asking them how they think other people will answer it. The idea being that maybe you can identify the "correct minority" if they're answering with some other answer, but correctly predicting the way in which most people will get it wrong, i.e., showing that they are aware of the trick that most people will fall prey to.
6/10/12
monty hall
I was talking to Christopher Lin at the University of Washington about Mechanical Turk experiments. One issue that came up is the problem of knowing the correct answer from a crowd when most people in the crowd are wrong. For instance, when answering a tricky brain teaser, one might imagine that most people are tricked, so the most popular answer is not correct. Yu An Sun has done some work on this, suggesting the tool of asking for answers in one pass, and then asking people to select from those answers in another pass. The idea is that people may not be tricked as easily if they have options to chose from, since seeing the correct answer may in some way unveil the trick.
I mentioned a similar problem with crowd sourcing which is trying to know if people are lying when it comes to subjective questions, where there is no ground truth at all. For instance, imagine asking people to say which logo design they like better. People may have legitimate differences of opinion. One tool I've heard of for combatting this is bayesian truth serum, which I've mentioned in some other blog post, but the basic idea is to ask people two questions: "what is your opinion?" and "what do you think most other people's opinion will be?".
Christopher then suggested the idea of applying this to the brain teaser problem: asking people for their answer on a brain teaser, and also asking them how they think other people will answer it. The idea being that maybe you can identify the "correct minority" if they're answering with some other answer, but correctly predicting the way in which most people will get it wrong, i.e., showing that they are aware of the trick that most people will fall prey to.
So, I asked 100 turkers to answer the Monty Hall problem, which is tricky, and to predict how other people would answer it.
I phrased it like this:
Imagine a game show. Everyday day, a person gets a chancee to win a prize. The prize is inside one of three boxes. The player always chooses a box, and then the game show host always reveals one of the remaining boxes to be empty, and allows the player to switch their choice to the remaining box if they want.
Should they switch their choice?
How will other people answer?
The raw results are here. Note that there are only 91 responses. I only waited 24 hours, and that's how many responses I got in that time. The column with the numbers represents work time in seconds. The "maybe" option refers to "It Doesn't Matter".
I mentioned a similar problem with crowd sourcing which is trying to know if people are lying when it comes to subjective questions, where there is no ground truth at all. For instance, imagine asking people to say which logo design they like better. People may have legitimate differences of opinion. One tool I've heard of for combatting this is bayesian truth serum, which I've mentioned in some other blog post, but the basic idea is to ask people two questions: "what is your opinion?" and "what do you think most other people's opinion will be?".
Christopher then suggested the idea of applying this to the brain teaser problem: asking people for their answer on a brain teaser, and also asking them how they think other people will answer it. The idea being that maybe you can identify the "correct minority" if they're answering with some other answer, but correctly predicting the way in which most people will get it wrong, i.e., showing that they are aware of the trick that most people will fall prey to.
So, I asked 100 turkers to answer the Monty Hall problem, which is tricky, and to predict how other people would answer it.
I phrased it like this:
Imagine a game show. Everyday day, a person gets a chancee to win a prize. The prize is inside one of three boxes. The player always chooses a box, and then the game show host always reveals one of the remaining boxes to be empty, and allows the player to switch their choice to the remaining box if they want.
Should they switch their choice?
- Yes, Switch
- No, Keep First Guess
- It Doesn't Matter
How will other people answer?
- Yes, Switch
- No, Keep First Guess
- It Doesn't Matter
The raw results are here. Note that there are only 91 responses. I only waited 24 hours, and that's how many responses I got in that time. The column with the numbers represents work time in seconds. The "maybe" option refers to "It Doesn't Matter".
Most people answered it wrong. The correct answer is "Yes, Switch". However, the people who answer "Yes, Switch" typically thought that other people would also answer "Yes, Switch", so that kindof goes against my theory.
It's possible that the people who answered "Yes, Switch" were just choosing the first option from both sets of radio buttons, but some of these people spent over a minute thinking about it, so I'm not sure.
I've re-asked the question, this time with no radio buttons, forcing people to actually type something in. I'm hoping that this will make it a little more clear who is cheating (if anyone is cheating).
UPDATE:
I ran it again with this question:
(100 turkers, $0.05/assignment)
Imagine there is a prize in one of three boxes. You chose one of the boxes, and then someone opens one of the other two boxes, revealing nothing inside. You then switch your guess to the other unopened box.
What percent chance do you have of this being the box with the prize?
[ ] %
What answer do you think most other people will give?
[ ] %
UPDATE:
I ran it again with this question:
(100 turkers, $0.05/assignment)
Imagine there is a prize in one of three boxes. You chose one of the boxes, and then someone opens one of the other two boxes, revealing nothing inside. You then switch your guess to the other unopened box.
What percent chance do you have of this being the box with the prize?
[ ] %
What answer do you think most other people will give?
[ ] %
The raw results are here.
The answers look pretty scattered, but most people (58 out of 100) put 50% for the first question, so we might consider only keeping people's answers who correctly guessed 50% for the second question. Doing this we get:
46 said 50%
3 said 66%
2 said 60%
1 said 66.66%
1 said 65%
1 said 33%
1 said 40%
1 said 25%
1 said 75%
The most common answer after 50% is now 66%, which is correct. I didn't tell people how accurate to be, so I suspect the people who said 60% and 65% also meant 66%.
In any case, this seems like a promising way to track down the correct answer from a crowd, even when most people get it wrong.
5/26/12
bayesian truth serum data
In a previous post, I took 100 yes/no questions and asked 20 turkers to answer each question. I also asked each turker to predict how other people would answer each question. [update: this is a refined version of this post]
That post also included an app where readers could answer questions themselves, and then see how turkers and other blog readers answered those questions.
At this point, there are about 40 answers for each question -- so about 20 answers from turkers, and 20 from blog-readers.
Here is an anonymized version of the data as JSON. The main object has three keys: users, questions, and answers*.
This plot shows people's average guesses as a function of the actual answers for a question. A point at X, Y on this plot means that for some question, X*100% of the people answered "yes", but the average guess people made about how many people would answer "yes" was Y*100%.
My main takeaway from this plot is that it seems like people are conservative in their guesses.
The next plot shows people's average guesses as a function of the actual answers for a question, grouped by how people answered the question, where blue is "yes" and red is "no". A red point on this plot at X, Y means that for some question, X*100% of the people answered "yes", but the average guess made by those who answered "no" was that Y*100% of the people would answer "yes".
I was about to write that people tend to over-predict that people will answer the same way they did, but that is not quite true. For instance, on questions where 90% of people say "yes", the people who say "yes" predict that only 70% will say "yes". Hence, I should instead say that people who answered "yes" to a question tend to predict that the outcome will have more yesses than predicted by those people who answered "no" to the same question.
If you run any analysis on the data, I'd love to hear about it! Here are some questions I'd like to know the answers to:
* Note that some of the answers have extra fields: otherQuestion, otherQuestionAbove, and otherQuestionOlder. These provide data about what other question was visible in the interface when the given answer was provided, and whether it appeared above the answered question, and whether it had been in the interface longer. These fields should only appear in instances where there was only one other question available to answer. Note that the interface begins by showing three questions simultaneously, but it pretends that the top questions are "older" for the purposes of otherQuestionOlder.
That post also included an app where readers could answer questions themselves, and then see how turkers and other blog readers answered those questions.
At this point, there are about 40 answers for each question -- so about 20 answers from turkers, and 20 from blog-readers.
Here is an anonymized version of the data as JSON. The main object has three keys: users, questions, and answers*.
This plot shows people's average guesses as a function of the actual answers for a question. A point at X, Y on this plot means that for some question, X*100% of the people answered "yes", but the average guess people made about how many people would answer "yes" was Y*100%.
The next plot shows people's average guesses as a function of the actual answers for a question, grouped by how people answered the question, where blue is "yes" and red is "no". A red point on this plot at X, Y means that for some question, X*100% of the people answered "yes", but the average guess made by those who answered "no" was that Y*100% of the people would answer "yes".
If you run any analysis on the data, I'd love to hear about it! Here are some questions I'd like to know the answers to:
- Are any questions correlated with each other? I'm not sure if there is enough data for this. We need instances where many people answered the same questions.
- Which questions are the most popular? One way to find out may be to look at the "otherQuestion" data*, e.g., a user had a choice between two questions to answer, and they answered this one instead of the "otherQuestion", so perhaps they like this one more.
- Are there any interesting differences between turkers and blog readers?
- How many questions did people answer? I'm sure this follows a power law.
- Are there any interesting results for specific questions?
* Note that some of the answers have extra fields: otherQuestion, otherQuestionAbove, and otherQuestionOlder. These provide data about what other question was visible in the interface when the given answer was provided, and whether it appeared above the answered question, and whether it had been in the interface longer. These fields should only appear in instances where there was only one other question available to answer. Note that the interface begins by showing three questions simultaneously, but it pretends that the top questions are "older" for the purposes of otherQuestionOlder.
bts data
Here's data about this experiment, having to do with bayesian truth serum. The data contains answers to 100 questions from both turkers and blog readers (all anonymized).
The questions were displayed three at a time, where usually one of the questions was showing the results, and there were two questions you could answer. In such cases:
I'm not sure these numbers are really meaningful though because one common case seems like it would be the case where there are 3 questions, the user answers the first one (which is not included in this data since there are 3 questions instead of 2), and then the user answers the second one. On the other hand, this question would be both on-top and older, so the bias would skew each figure the same amount, meaning that the difference in percentages probably is meaningful (i.e. older-ness of question is more predictive than which question is on top).
Graph 1: actual vs mean-guess
Graph 2: actual vs mean-guess for people who answered 'yes' and 'no'
Here's the Excel file for the above plots.
JavaScript Eval code (after pasting this data into the lower-left textarea):
var db = eval(input)
foreach(db.questions, function (q) {
q.answers = {}
q.guesses = []
})
foreach(db.answers, function (a) {
var q = db.questions[a.question]
q.answers[a.text] = q.answers[a.text] ? q.answers[a.text] + 1 : 1
if (a.text.match(/yes|no/))
q.guesses.push(a.guess)
})
foreach(db.questions, function (q) {
var s = ""
s += q.answers.yes / (q.answers.yes + q.answers.no)
s += ','
var sum = 0
var total = 0
foreach(q.guesses, function (g) { sum += g; total += 1 })
s += sum / total
print(s)
})
The questions were displayed three at a time, where usually one of the questions was showing the results, and there were two questions you could answer. In such cases:
- 57.2% of the time, people answered the top question
- 76.4% of the time, people answered the older question (the one that had been there longer)
I'm not sure these numbers are really meaningful though because one common case seems like it would be the case where there are 3 questions, the user answers the first one (which is not included in this data since there are 3 questions instead of 2), and then the user answers the second one. On the other hand, this question would be both on-top and older, so the bias would skew each figure the same amount, meaning that the difference in percentages probably is meaningful (i.e. older-ness of question is more predictive than which question is on top).
Graph 1: actual vs mean-guess
Graph 2: actual vs mean-guess for people who answered 'yes' and 'no'
Here's the Excel file for the above plots.
JavaScript Eval code (after pasting this data into the lower-left textarea):
var db = eval(input)
foreach(db.questions, function (q) {
q.answers = {}
q.guesses = []
})
foreach(db.answers, function (a) {
var q = db.questions[a.question]
q.answers[a.text] = q.answers[a.text] ? q.answers[a.text] + 1 : 1
if (a.text.match(/yes|no/))
q.guesses.push(a.guess)
})
foreach(db.questions, function (q) {
var s = ""
s += q.answers.yes / (q.answers.yes + q.answers.no)
s += ','
var sum = 0
var total = 0
foreach(q.guesses, function (g) { sum += g; total += 1 })
s += sum / total
print(s)
})
5/25/12
iterative image descriptions revisited, again
A while ago, I ran an experiment on mturk involving people iteratively improving image descriptions.
If we want to build a model of the steps involved in this process, we might consider three things:
Here is the data for M (each row has two number X, Y, where someone took a description of quality X and wrote a description of quality Y)
Here is the data for V (each row has three numbers, X, Y, Z, where someone saw a description of quality X and Y, and they voted for the "correct" one iff Z = 1)
Here's a histogram plot of I (R command: hist(I$rating, breaks=0:10) )
Here's a sortof 2d histogram (looking down) of M (thanks to Panos Ipeirotis for these R commands):
> library(KernSmooth)
> m <- read.csv("M.txt")
> model <- bkde2D(m, bandwidth=c(2,2), gridsize=c(200, 200), truncate=TRUE)
> filled.contour(model$x1, model$x2, model$fhat, xlim=range(c(1,10)), ylim=range(c(1,10)), nlevels=50, col=terrain.colors(70, alpha =1.0))
Here's an even better plot from Panos Ipeirotis:
> library(som)
> m <- read.csv("M.txt")
> model <- bkde2D(m, bandwidth=c(dpik(m$r1),dpik(m$r2)), gridsize=c(150,
150), truncate=TRUE)
> filled.contour(model$x1, model$x2, normalize(model$fhat, byrow=TRUE),
xlim=range(c(3,10)), ylim=range(c(3,10)), nlevels=40,
col=terrain.colors(50, alpha =1.0))
If we want to build a model of the steps involved in this process, we might consider three things:
- I : the distribution of description qualities when someone writes a description from scratch.
- M : the distribution of description qualities when someone writes a description based on a description of some quality.
- V : the probability of a voter correctly picking the description of higher quality given two descriptions with some difference in quality (presumably the probability would be 0.5 if the descriptions had equal quality).
Here is the data for M (each row has two number X, Y, where someone took a description of quality X and wrote a description of quality Y)
Here is the data for V (each row has three numbers, X, Y, Z, where someone saw a description of quality X and Y, and they voted for the "correct" one iff Z = 1)
Here's a histogram plot of I (R command: hist(I$rating, breaks=0:10) )
Here's a sortof 2d histogram (looking down) of M (thanks to Panos Ipeirotis for these R commands):
> library(KernSmooth)
> m <- read.csv("M.txt")
> model <- bkde2D(m, bandwidth=c(2,2), gridsize=c(200, 200), truncate=TRUE)
> filled.contour(model$x1, model$x2, model$fhat, xlim=range(c(1,10)), ylim=range(c(1,10)), nlevels=50, col=terrain.colors(70, alpha =1.0))
Here's an even better plot from Panos Ipeirotis:
> library(som)
> m <- read.csv("M.txt")
> model <- bkde2D(m, bandwidth=c(dpik(m$r1),dpik(m$r2)), gridsize=c(150,
150), truncate=TRUE)
> filled.contour(model$x1, model$x2, normalize(model$fhat, byrow=TRUE),
xlim=range(c(3,10)), ylim=range(c(3,10)), nlevels=40,
col=terrain.colors(50, alpha =1.0))
TurkPad
I created something called TurkPad, which is essentially just Etherpad, but with the ability to bring in turk workers. And it's free (for you). People can spend up to $1 for each TurkPad instance. The site spends my own turk money, and reveals my turk account balance.
There is also a PayPal link for people to donate money, with the vague hope that people will donate money to cover the cost of the experiments they run. I could make people pay with PayPal to start each experiment, but then it would have more friction, and people might complain to me saying: "Hey, I paid you money, and I didn't get what I wanted!"
Here are the results from several early trails (I forget what the people/rate's were for each one, but I give my guess):
Brainstorming a company name (I think 10 people at $0.05/task):
John Horton suggested this task: "Come up with questions to ask applicants to a data entry job" (I think 20 people at $0.05)
Panos Ipeirotis suggested this task (I think 10 people at $0.05):
Coin flipping (I think 10 people at $0.01/task):
There is also a PayPal link for people to donate money, with the vague hope that people will donate money to cover the cost of the experiments they run. I could make people pay with PayPal to start each experiment, but then it would have more friction, and people might complain to me saying: "Hey, I paid you money, and I didn't get what I wanted!"
Here are the results from several early trails (I forget what the people/rate's were for each one, but I give my guess):
Brainstorming a company name (I think 10 people at $0.05/task):
original typewith.me pad: http://typewith.me/p/DgLMG36g
John Horton suggested this task: "Come up with questions to ask applicants to a data entry job" (I think 20 people at $0.05)
original PiratePad: http://piratepad.net/zkVypZgPFb
[...some blank lines...]
[...some blank lines...]
[...some blank lines...]
1. Have you done data entry job before
[...some blank lines...]
1.What is your typing speed
Panos Ipeirotis suggested this task (I think 10 people at $0.05):
original PiratePad here: http://piratepad.net/DwGZbiGN22
Coin flipping (I think 10 people at $0.01/task):
original typewith.me pad: http://typewith.me/p/DgLMG36gdE
5/24/12
flipping coins
When you ask a turker to flip a coin, it will land on heads about 2/3 of the time. There is an interesting series of blog posts about this: Coin Flipping, Coin Flipping with Bonuses, Coin Flips Revisited, and even more recently Michael Bernstein repeated the experiment with 1000 turkers and got 687 heads.
So when Google consumer surveys came out, I asked Allison Moberger to post a question for me: "Please flip an actual coin and choose the result below," with the options of "Heads" and "Tails".
We went for the smallest experiment allowed by Google, which is 1000 people, at $0.10 per question, so $100.
The result: 65.9% heads.
I didn't realize this, but even though you pay for 1000 people, Google doesn't have demographic information on everyone who answers, and it throws out people it doesn't know about. Hence, this 65.9% is from 794 'respondents with demographics', and I'm not sure how to see what the remaining 206 people did. [EDIT: Jack Hebert says "To see all 1000 responses, click on the 'gear' icon then set 'Prefer weighted' to Off." He's right. Thanks Jack! He also mentioned that I can share the data. Apparently it is public by default, so here's the link.]
The demographics are cool though, and they automatically look for statistically significant differences among the demographics, called "Insights". It says "124 insights investigated. 2.5 false discoveries expected on average." There were indeed two insights, so they're probably both false: "Among women, those in the US South picked Heads more than those in the US Northeast." and "Among people in the US South, women picked Heads more than men."
We went for the smallest experiment allowed by Google, which is 1000 people, at $0.10 per question, so $100.
The result: 65.9% heads.
I didn't realize this, but even though you pay for 1000 people, Google doesn't have demographic information on everyone who answers, and it throws out people it doesn't know about. Hence, this 65.9% is from 794 'respondents with demographics', and I'm not sure how to see what the remaining 206 people did. [EDIT: Jack Hebert says "To see all 1000 responses, click on the 'gear' icon then set 'Prefer weighted' to Off." He's right. Thanks Jack! He also mentioned that I can share the data. Apparently it is public by default, so here's the link.]
The demographics are cool though, and they automatically look for statistically significant differences among the demographics, called "Insights". It says "124 insights investigated. 2.5 false discoveries expected on average." There were indeed two insights, so they're probably both false: "Among women, those in the US South picked Heads more than those in the US Northeast." and "Among people in the US South, women picked Heads more than men."
bayesian truth serum
I took 100 yes/no questions and asked 20 turkers to answer each question. I also asked each turker to predict how other people would answer each question.
You can see these questions to the right. If you answer one, and predict how other people will answer it, then you can see how the turkers and other blog readers answered the question. You also get points: 100 points if your prediction is exactly right, and it goes down to 0 when you are 50 points away from the right percentage. New questions will come when you answer these questions.
Sometimes you'll see that the average answer and average prediction are pretty different, suggesting cases where people have a distorted view of reality. I'd like to make the data public and say something interesting about it, but first I want to let people make guesses, and try to get a good score, without already knowing the answers.
For me, the idea of asking people to predict what other people will do comes from 'bayesian truth serum', though that paper proposes some fancy math which I am not using here. In particular, bayesian truth serum gives you a score based not only on your prediction, but also on your answer, which would seem like it incentivizes people to answer dishonestly so as to maximize their score, but for some math reason, they claim it doesn't. Here, I'm just giving a score based on your prediction, which is simpler to understand.
This idea also came up again for me when listening to a talk by Justin Wolfers at the Collective Intelligence conference about forecasting elections. Wolfers has some interesting data showing that asking 'who do you think will win' is more predictive than 'who will you vote for', which sortof makes sense since you're sortof asking people to take the average vote of all the people they know, which may effecitvely increase your sample size; however, the effect is stronger than I thought, since he also has data showing that asking a biased sample of people, like all republicans, 'who do you think will win?' is better than asking an unbiased sample of people 'who will you vote for?'.
Anyway, long store short, I wanted to play around with the idea of asking 'how will other people answer this question?'. Hopefully a lot of people answer the questions in this prototype, and I can write a subsequent blog post about that data.
You can see these questions to the right. If you answer one, and predict how other people will answer it, then you can see how the turkers and other blog readers answered the question. You also get points: 100 points if your prediction is exactly right, and it goes down to 0 when you are 50 points away from the right percentage. New questions will come when you answer these questions.
Sometimes you'll see that the average answer and average prediction are pretty different, suggesting cases where people have a distorted view of reality. I'd like to make the data public and say something interesting about it, but first I want to let people make guesses, and try to get a good score, without already knowing the answers.
For me, the idea of asking people to predict what other people will do comes from 'bayesian truth serum', though that paper proposes some fancy math which I am not using here. In particular, bayesian truth serum gives you a score based not only on your prediction, but also on your answer, which would seem like it incentivizes people to answer dishonestly so as to maximize their score, but for some math reason, they claim it doesn't. Here, I'm just giving a score based on your prediction, which is simpler to understand.
This idea also came up again for me when listening to a talk by Justin Wolfers at the Collective Intelligence conference about forecasting elections. Wolfers has some interesting data showing that asking 'who do you think will win' is more predictive than 'who will you vote for', which sortof makes sense since you're sortof asking people to take the average vote of all the people they know, which may effecitvely increase your sample size; however, the effect is stronger than I thought, since he also has data showing that asking a biased sample of people, like all republicans, 'who do you think will win?' is better than asking an unbiased sample of people 'who will you vote for?'.
Anyway, long store short, I wanted to play around with the idea of asking 'how will other people answer this question?'. Hopefully a lot of people answer the questions in this prototype, and I can write a subsequent blog post about that data.
talent court
Summary
Check out TalentCourt.Introduction
This post expands upon the ideas of the quality game post. It is an idea dump that I'd like to turn into a paper at some point (note that some of this 'dump' is written by Allison Moberger, who by coincidence is currently at the top of TalentCourt writing [or was, when this post was originally written], so the bad writing here is probably mine).Motivation
In short, we want real-time access to experts to perform micro-tasks. For example, we might want to hire an expert JavaScript programmer for the next 10 minutes to help write a utility method. Here are some example use-cases:Logo design: Imagine designing a logo in the following way: pay 20 expert sketch artists $1 each for a 5 minute sketch. Then use traditional crowdsourcing to find the best 5 sketches. Then pay 5 expert photoshop artists $5 each for a 20 minute mockup of the best sketches. Then use traditional crowdsourcing to find the best mockup, and pay 1 expert designer $20 for 1 hour to put the finishing touches on the best mockup. Now we've paid a bit over $100 for a logo. It would be interesting to see how this logo compares to what is generated for a similar amount on a site like 99 designs. The advantage of this approach is that nobody works for no pay, which may be more efficient, e.g., more quality per dollar.
Micro-outsourcing: Max Goldman has built a system that supports real-time collaborative programming called Collabode. One use-case mentioned in the Collabode paper is micro-outsourcing, where a main programmer delegates tasks, like filling in method bodies of a new class, in real-time as part of their flow. It would be interesting to see if this style of programming actually works. One could also imagine a similar working style for writing, where a main writer writes an outline for paragraphs of a paper, and then hires expert writers in real-time to flesh-out the outliness. This might allow the writer to spend more time thinking at a higher level about the overal ideas and organization of the paper, and "directing" the creation of the paper.
Self-repairing web services: Imagine that you have written a web service that periodically grabs data from one source (e.g. Rhapsody), munges it, and sends it to another service (e.g. last.fm). Now imagine that the data source changes the format of their data, which breaks some regular expressions in the parsing script, raising an exception. It might be cool if the system could handle the exception by automatically hiring an expert to repair the script.
Problem
The essential problem is this: if you are going to hire someone in real-time, you need some very fast method of determining whether they are qualified for your job, and currently there is no such method. There are lots of systems that try to do this, but they all fall short in some way:Standardized tests: The most commonly used method of identifying skilled applicants is the standardized test. Nearly everyone has been subject to this, as the use of tests like the SAT, ACT, GRE, MCAT, and LSAT are widespread throughout all levels of the academic community. If an applicant performs poorly on a standardized test, the admissions office can simply dismiss the application and move on to others. However, there are downsides to standardized testing: they're expensive to create and administer, difficult to keep secret, and easily-gameable. Rather than learning and understanding all of the math or language through life experience, many simply take classes to study specifically for the test, and this could cause an applicant to receive a high score even though their knowledge is lacking. For online tests, cheating is even harder to prevent. For example, oDesk has standardized tests for many skills, but when I type "odesk java test" into Google, it auto-suggests "odesk java test answers", and the top link has the answers.
Ratings: Sites like eBay and oDesk allow users to rate prior transactions. On oDesk especially, this can create a "cold-start" problem, where one needs work to get a rating, but needs a good rating to get work. Ratings also have an added problem of "grade inflation"; as anything less than the best rating could hurt a candidate's future prospects, employers feel some pressure to hand out the best rating unless something went very bad. They are also static; a past employer's ratings tell a new potential employer nothing about any skills a candidate has gained since they last worked for the past employer. Finally, ratings do not include a notion of how hard a task was, or how closely related it is to the current employers needs.
Portfolio: Artists and designers often present a portfolio of their work that employers can use to judge their skill and style. Unfortunately, portfolios can also be gamed — people can post work that they didn't do. A more suble and less intentional way of cheating is for someone to include work in a portfolio that they contributed to, without explaining exactly what they did. This is understandable in a close collaboration, where it is difficult to tease apart who contributed what, but the net effect in any case is that portfolios cannot always be trusted.
Interviews: Another option, frequently used by technical employers, is the expert interview; that is, applicants have a face-to-face interview with an expert in the field, whose knowledge allows them to evaluate the skill of the applicants. This also tends to be gameable to some degree, as a large portion of the internet seems devoted to preparing interviewees to answer "Microsoft questions"; it also has a problem of scale, as the time and energy required to find the best candidates increases to prohibitive levels very quickly relative to the pool size. Finally, it also has another flaw at the meta-assessment level: how does one identify an expert interviewer, except through an interview with another expert? There is no good way to "assess the assessors".
The last two techniques are not really suitable for real-time hiring anyway, because they are subjective, and generally require a human to spend time converting the material in the portfolio or interview into a yes/no decision about whether to hire them. In practice, this assessment time can take more time than a micro-task itself.
Solution
Our essential idea is to build competitive games around different skills, and use an elo-style rating system to encode the skill of the players. One way to think of these games is as games with a purpose, where the purpose of each game is to evaluate the skill of the players. Depending on the skill, the game may involve players doing some or all of the following:- generating questions
- answering questions
- comparing answers
- By generating random questions, or having players generate new questions as part of the game, we can afford to keep the lifetime of questions very small, to mitigate the danger of people posting "cheat sheets" online.
- By comparing questions, we can guard against needing to know how difficult a question is, since it will be the same difficulty for both players, and each player just needs to answer it better than their opponent.
- By having humans evaluate questions, we allow for "essay questions", or questions with subjective answers, which guards against many issues with multiple choice questions.
- By making each contest small, we allow players to increase their score without blocking out a large time in their schedule (e.g. 5 minutes games as opposed to a 30min or 1hour test).
- Because of the dynamic nature of the elo rating system, it notices as a player increases their skill (as opposed to standardized tests, which may require people to wait a month or a year before taking a test again.. presumably to prevent people from simply remembering the questions from the last time).
- Again because of the nature of the elo rating system, a player may not need to play many games before the system has a good idea how skilled they are (similar to the adaptive questions in the GRE).
- Because scores are relative to other people, the elo rating system can differentiate between people over a very broad range of skill levels, as opposed to standardized tests like the GRE math test, where many people get a perfect score, and the test has no ability to discriminate between these people.
TalentCourt currently includes a Writing and Drawing game. Each game proceeds as follows: two users are shown a random prompt (3 random words for writing, and 1 random word for drawing). Each user then has 5 minutes to write a short passage using all three of the words in some meaningful way, or draw the given word. Then three different people are asked to select the best passage or sketch. We previously gave voters instructions like choose "the most natural sounding paragraph", but the current version has no instructions for voting. The passage with the most votes wins, and this user's score increases, while the loser's score decreases. Scores are updated according to the elo rating system (and will probably move to the TrueSkill algorithm soon).
Cheating
We want to use the scores from these games for hiring decisions, which provides a lot of incentive to cheat. We designed the games to mitigate various methods of cheating:Perhaps the most obvious way to cheat at either game is to use Google. In the English game, one could enter the three words as a query to try to find a paragraph someone else has written using the words. The search results tend to be webpages that contain all 3 words without all the words being close enough together to use as input in the game. Hopefully, under time pressure, finding a short passage with all 3 words in a sensible context is more difficult than writing one from scratch. If it becomes a problem, we could also use four words instead of three.
Similarly, in the Sketch game, one could search the word and find an image, but the data uploaded to the server includes a sequence of painting strokes (rather than the raw pixel data), making it difficult to cheat by uploading a pre-drawn image. This does not guard against the possibility of a user tracing an image or drawing using another image as a reference. (Note: actually, the current implementation does upload the raw image data in addition to the strokes, but this is a temporary work-around for some technical issues.)
A different way to cheat would be to game the votes, rather than the inputs. It is possible for users to have friends, or other accounts, vote for their entries; to help prevent this, we do not reveal the authors of each entry to voters until after their votes have been cast, but there is still the possibility that users could communicate outside the system. We also do not let users choose which contests to vote on, so in a liquid market, the likelihood of voting on a friend's work would hopefully be small.
A related way to cheat would be to have confederate competitors deliberately perform poorly. The design of the game makes this difficult for a number of reasons: users do not get to choose who to compete against, and are not told who they are competing against until the voting on their input is closed; the elo rating system also makes it difficult to gain substantial rating increases from this method, as large score increases are gained from defeating superior opponents, not those who perform poorly.
Future Work
What ideas do we have for the future?Programming Game: There are two ways this might work:
Way 1: Do it similar to the writing game, but instead of words, we use methods from the standard API of the language being tested, and ask people to write a short program that uses all 3 methods in some meaningful way.
Way 2: Have people generate programming interview questions. This idea is less baked — it is not clear whether people will be good at coming up with questions, so our first tests will involve just asking programmers to come up with questions, to get an idea of what sorts of questions they'll ask.
Graphic Design Game: This could be similar to the sketch game, where there is an HTML-based drawing tool, except this one might include a set of shapes and text that the designer is allowed to move/scale/rotate and set the color of. The prompt might also include a word like "peaceful", "energetic", or "efficient" that the design should convey.
quality game
Problem: say we have a million people, and we want to assess each person's skill at X, so that we know who to hire.
Solution: we're experimenting with a game with a purpose, the purpose of which is to determine people's skill.
For instance, for assessing English ability, we tried the following game: two users are shown three random words. Each user has 5 minutes to write a short passage using all three of the words in some meaningful way. Here is the interface:
The words are chosen from a list of 850 "basic English words". We wanted common words that every English speaker would be likely to know, since we were less interested in testing people's vocabulary, and more interested in testing people's ability to use words together naturally.
We deployed a prototype on App Engine. For the first week, we tried to attract users to it by posting a job on oDesk offering $20 to whoever had the highest score each night at midnight Pacific Time. We also released the link to some friends (note that we still paid the highest oDesk worker, even if one of our friends was actually at the top of the leader board).
For the words "able", "trick" and "sand", here are two entries from high scoring players:
For the words "cow", "star" and "living", here are two entries from lower scoring players:
Incidentally, the right-hand passage won all 3 votes for that last contest. You can see all the data here.
Solution: we're experimenting with a game with a purpose, the purpose of which is to determine people's skill.
For instance, for assessing English ability, we tried the following game: two users are shown three random words. Each user has 5 minutes to write a short passage using all three of the words in some meaningful way. Here is the interface:
Then three different people are asked to select "the most natural sounding paragraph". The passage with the most votes wins, and this user's score increases, while the loser's score decreases. Scores are updated according to the elo rating system, and they start out at 1500.
The words are chosen from a list of 850 "basic English words". We wanted common words that every English speaker would be likely to know, since we were less interested in testing people's vocabulary, and more interested in testing people's ability to use words together naturally.
We deployed a prototype on App Engine. For the first week, we tried to attract users to it by posting a job on oDesk offering $20 to whoever had the highest score each night at midnight Pacific Time. We also released the link to some friends (note that we still paid the highest oDesk worker, even if one of our friends was actually at the top of the leader board).
For the words "able", "trick" and "sand", here are two entries from high scoring players:
I've never been able to trick my older sister into eating sand. I suppose it's just as well. Who knows what she would have done in retaliation!
The trick to building sand castles is getting the exact ratio between sand and water. Too little or too much sand, and the castle will be unable to stand.
There's a cow living in our barn. Just like a knight he wakes up all night only to watch the star that light.
Do you think that sometimes, somewhere, a lonely cow looks at the stars and wonders what's the point in living?
Subscribe to:
Posts (Atom)