Stanford InfoLab Publication Server

Quality Control for Comparison Microtasks

Venetis, Petros and Garcia-Molina, Hector (2012) Quality Control for Comparison Microtasks. In: CrowdKDD 2012, August 12, 2012, Beijing, China.


PDF - Published Version


We study quality control mechanisms for a crowdsourcing system where workers perform object comparison tasks. We study error masking techniques (e.g., voting) and detection of bad workers. For the latter, we consider using gold-standard questions, as well as disagreement with the plurality answer. We perform experiments on Mechanical Turk that yield insights as to the role of task difficulty in quality control, and the effectiveness of the schemes.

Item Type:Conference or Workshop Item (Paper)
Uncontrolled Keywords:crowdsourcing, comparisons, comparison microtasks, quality, quality control, masking, detection, task difficulty, worker evaluation
Related URLs:Author Homepage
ID Code:1044
Deposited By:Petros Venetis
Deposited On:26 Jun 2012 12:04
Last Modified:26 Jun 2012 12:04

Download statistics

Repository Staff Only: item control page