quality game

Problem: say we have a million people, and we want to assess each person's skill at X, so that we know who to hire.

Solution: we're experimenting with a game with a purpose, the purpose of which is to determine people's skill.

For instance, for assessing English ability, we tried the following game: two users are shown three random words. Each user has 5 minutes to write a short passage using all three of the words in some meaningful way. Here is the interface:

Then three different people are asked to select "the most natural sounding paragraph". The passage with the most votes wins, and this user's score increases, while the loser's score decreases. Scores are updated according to the elo rating system, and they start out at 1500.

The words are chosen from a list of 850 "basic English words". We wanted common words that every English speaker would be likely to know, since we were less interested in testing people's vocabulary, and more interested in testing people's ability to use words together naturally.

We deployed a prototype on App Engine. For the first week, we tried to attract users to it by posting a job on oDesk offering $20 to whoever had the highest score each night at midnight Pacific Time. We also released the link to some friends (note that we still paid the highest oDesk worker, even if one of our friends was actually at the top of the leader board).

For the words "able", "trick" and "sand", here are two entries from high scoring players:

I've never been able to trick my older sister into eating sand. I suppose it's just as well. Who knows what she would have done in retaliation!

The trick to building sand castles is getting the exact ratio between sand and water. Too little or too much sand, and the castle will be unable to stand.

For the words "cow", "star" and "living", here are two entries from lower scoring players:

There's a cow living in our barn. Just like a knight he wakes up all night only to watch the star that light.

Do you think that sometimes, somewhere, a lonely cow looks at the stars and wonders what's the point in living?

Incidentally, the right-hand passage won all 3 votes for that last contest. You can see all the data here.