Rondo a la Mechanical Turk*

Last week I commented on a classic older paper on crowdsourcing from 2009 (“Soylent”. It’s people.)

In the intervening years, we have accumulated experience with various permutations of the crowdsourcing idea, which is leading to a refined understanding of how it works and where it breaks.

Last month, Ujwal Gadiraju
and colleagues reviewed what we have learned about “paid microtask crowdsourcing”, i.e., a la Mechanical Turk [1]. (There are many variants in this general idea, see Quinn (2011) [2].) This article surveys what is being done with this type of crowdsourcing [1].

They report frequent types of tasks microtasked, including clever things (verification and validation, detection of bots, checking web content), audio transcription and stupid things (“polls”, and even “just watch this”). The latter sound like marketing research to me, though they would also be phishing opportunities.

Examining the ten years of activity on Mechanical Turk, they note that traffic has increased to the level of 7,500 tasks completed per hour (world wide). This is actually much less than I expected, but MTurk isn’t the only game in town so the total is larger. They note that payments have trended up, though they are still small amounts for small tasks.

The “crowd” of workers is diverse and often includes dishonest people who are out to game the system. This is an obvious hazard of the impersonal, incentive based system. For example, responding as fast as possible with random answers might accrue payments with little effort by the worker and no gain by the requester. A common technique for detecting low performance are check questions that have obvious but unfakable answers (which the authors term “gold standard test questions”).

Given this uncertainty about the workers and the quality of work, research has explored ways to evaluate and control the performance of crowdworker. One goal is to motivate and retain good crowdworkers through incentives (and “gamificatiion”) and making sure they work on important tasks. One the other side of that coin, research has looked for automated monitoring systems, to detect poor or malicious behavior and “route around” it.

Finally, G. and colleagues note some ethical perspectives, which, amazingly enough, require looking at things from the perspective of the workers. Paying the lowest possibly fees for tiny periods of time may be “efficient”, but it isn’t really good for most workers. How should payments be set? How can crowdworkers survive long term? And if crowdworkers contribute (in their own small way) to producing wealth, then they deserve more than a few pennies of compensation, no?


 

*See Brubeck (1959).

  1. Gadiraju, Ujwal, Gianluca Demartini, Ricardo Kawase, and Stefan Dietze, Human Beyond the Machine: Challenges and Opportunities of Microtask Crowdsourcing. Intelligent Systems, IEEE, 30 (4):81-85, 2015.
  2. Quinn, Alexander J. and Benjamin B. Bederson, http://alexquinn.org/papers/Human%20Computation,%20A%20Survey%20and%20Taxonomy%20of%20a%20Growing%20Field%20%28CHI%202011%29.pdf, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2011, ACM: Vancouver, BC, Canada. p. 1403-1412.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s