Stanford InfoLab Publication Server

Globally Optimal Crowdsourcing Quality Management

Das Sarma, Akash and Parameswaran, Aditya and Widom, Jennifer Globally Optimal Crowdsourcing Quality Management. Technical Report. Stanford InfoLab.


PDF - Submitted for Publication


We study crowdsourcing quality management, that is, given worker responses to a set of tasks, our goal is to jointly estimate the true answers for the tasks, as well as the quality of the workers. Prior work on this problem relies primarily on applying Expectation-Maximization (EM) on the underlying maximum likelihood problem to estimate true answers as well as worker quality. Unfortunately, EM only provides a locally optimal solution rather than a globally optimal one. Other solutions to the problem (that do not leverage EM) fail to provide global optimality guarantees as well. In this paper, we focus on filtering, where tasks require the evaluation of a yes/no predicate, and rating, where tasks elicit integer scores from a finite domain. We design algorithms for finding the global optimal estimates of correct task answers and worker quality for the underlying maximum likelihood problem, and characterize the complexity of these algorithms. Our algorithms conceptually consider all mappings from tasks to true answers (typically a very large number), while leveraging two key ideas to reduce, by several orders of magnitude, the number of mappings under consideration. We also demonstrate that these algorithms often find more accurate estimates than EM-based algorithms. This paper makes an important contribution towards understanding the inherent complexity of globally optimal crowdsourcing quality management.

Item Type:Techreport (Technical Report)
ID Code:1111
Deposited By:Akash Das Sarma
Deposited On:30 Nov 2014 16:22
Last Modified:30 Nov 2014 16:22

Download statistics

Repository Staff Only: item control page