Stanford InfoLab Publication Server

CrowdDQS: Dynamic Question Selection in Crowdsourcing Systems

Khan, Asif R. and Garcia-Molina, Hector (2016) CrowdDQS: Dynamic Question Selection in Crowdsourcing Systems. Technical Report. Stanford InfoLab.


This is the latest version of this item.

[img]PDF - Accepted Version


In this paper, we present CrowdDQS, a system that uses the most recent set of crowdsourced voting evidence to dynamically issue questions to workers on Amazon Mechanical Turk (AMT). CrowdDQS posts all questions to AMT in a single batch, but delays the decision of the exact question to issue a worker until the last moment, concentrating votes on uncertain questions to maximize accuracy. Unlike previous works, CrowdDQS also (1) optionally can also decide when it is more beneficial to issue gold standard questions with known answers than to solicit new votes (both can help us estimate worker accuracy, but gold standard questions provide a less noisy estimate of worker accuracy at the expense of not obtaining new votes), (2) estimates worker accuracies in real-time even with limited evidence (with or without gold standard questions), and (3) infers the distribution of worker skill levels to actively block poor workers. We deploy our system live on AMT to over 1000 crowdworkers, and find that using our techniques, CrowdDQS can accurately answer questions using up to 6x fewer votes than standard approaches. We also find there are many non-obvious practical challenges involved in deploying such a system seamlessly to crowdworkers, and discuss techniques to overcome these challenges.

Item Type:Techreport (Technical Report)
ID Code:1152
Deposited By:Asif Khan
Deposited On:21 Mar 2017 02:41
Last Modified:21 Mar 2017 02:41

Available Versions of this Item

Download statistics

Repository Staff Only: item control page