Venetis, Petros and Garcia-Molina, Hector (2012) Quality Control for Comparison Microtasks. In: CrowdKDD 2012, August 12, 2012, Beijing, China.
|PDF - Published Version|
We study quality control mechanisms for a crowdsourcing system where workers perform object comparison tasks. We study error masking techniques (e.g., voting) and detection of bad workers. For the latter, we consider using gold-standard questions, as well as disagreement with the plurality answer. We perform experiments on Mechanical Turk that yield insights as to the role of task difficulty in quality control, and the effectiveness of the schemes.
|Item Type:||Conference or Workshop Item (Paper)|
|Uncontrolled Keywords:||crowdsourcing, comparisons, comparison microtasks, quality, quality control, masking, detection, task difficulty, worker evaluation|
|Related URLs:||Author Homepage||http://i.stanford.edu/~venetis/|
|Deposited By:||Petros Venetis|
|Deposited On:||26 Jun 2012 12:04|
|Last Modified:||26 Jun 2012 12:04|
Repository Staff Only: item control page