Das Sarma, Akash and Parameswaran, Aditya and Widom, Jennifer Optimal Worker Quality and Answer Estimates in Crowd-Powered Filtering and Rating. In: HCOMP 2014, Work in Progress Track.
|PDF (HCOMP Work in Progress Paper) - Accepted Version|
We consider the problem of optimally filtering (or rating) a set of items based on predicates (or scoring) requiring human evaluation. Filtering and rating are ubiquitous problems across crowdsourcing applications. We consider the setting where we are given a set of items and a set of worker responses for each item: yes/no in the case of filtering and an integer value in the case of rating. We assume that items have a true inherent value that is unknown, and workers draw their responses from a common, but hidden, error distribution. Our goal is to simultaneously assign a ground truth to the item-set and estimate the worker error distribution. Previous work in this area has focused on heuristics such as Expectation Maximization (EM), providing only a local optima guarantee, while we have developed a general framework that finds a maximum likelihood solution. Our approach extends to a number of variations on the filtering and rating problems.
|Item Type:||Conference or Workshop Item (Paper)|
|Deposited By:||Akash Das Sarma|
|Deposited On:||19 Aug 2014 12:06|
|Last Modified:||19 Aug 2014 12:06|
Repository Staff Only: item control page