Christopher Mims, in technology review:

Amazon's Mechancial Turk is the ultimate in nearly anonymous outsourcing: any task that can be completed online can be accomplished by the combination of automated marketplace and human labor. Those who sign up to complete tasks - Turkers - are paid wages as low as pennies per chore to do everything from data entry to folk art.

Mechanical Turk is designed to complete tasks that are easy for humans and hard for machines, such as categorizing or identifying the content of images. The problem for Amazon and all its imitators, however, is that machines are getting better at many tasks, while the humans on Mechanical Turk, for reasons I'll explore in tomorrow's post, are getting worse.

Recently, for example, researchers working at the online review site Yelp released a paper (pdf) on their experience matching thousands of Mechanical Turkers against a supervised learning algorithm.

The results weren't pretty: in order to find a population of Turkers whose work was passable, the researchers first used Mechanical Turk to administer a test to 4,660 applicants. It was a multiple choice test to determine whether or not a Turker could identify the correct category for a business (Restaurant, Shopping, etc.) and verify, via its official website or by phone, its correct phone number and address.

79 passed. This was an extremely basic multiple choice test. It makes one wonder how the other 4,581 were smart enough to operate a web browser in the first place.

via Technology Review.