Stanford InfoLab Publication Server

Optimal Worker Quality and Answer Estimates in Crowd-Powered Filtering and Rating

Das Sarma, Akash and Parameswaran, Aditya and Widom, Jennifer Optimal Worker Quality and Answer Estimates in Crowd-Powered Filtering and Rating. In: HCOMP 2014, Work in Progress Track.


PDF (HCOMP Work in Progress Paper) - Accepted Version


We consider the problem of optimally filtering (or rating) a set of items based on predicates (or scoring) requiring human evaluation. Filtering and rating are ubiquitous problems across crowdsourcing applications. We consider the setting where we are given a set of items and a set of worker responses for each item: yes/no in the case of filtering and an integer value in the case of rating. We assume that items have a true inherent value that is unknown, and workers draw their responses from a common, but hidden, error distribution. Our goal is to simultaneously assign a ground truth to the item-set and estimate the worker error distribution. Previous work in this area has focused on heuristics such as Expectation Maximization (EM), providing only a local optima guarantee, while we have developed a general framework that finds a maximum likelihood solution. Our approach extends to a number of variations on the filtering and rating problems.

Item Type:Conference or Workshop Item (Paper)
ID Code:1098
Deposited By:Akash Das Sarma
Deposited On:19 Aug 2014 12:06
Last Modified:19 Aug 2014 12:06

Download statistics

Repository Staff Only: item control page