Stanford InfoLab Publication Server

CrowdScreen: Algorithms for Filtering Data with Humans

Parameswaran, Aditya and Garcia-Molina, Hector and Park, Hyunjung and Polyzotis, Neoklis and Ramesh, Aditya and Widom, Jennifer CrowdScreen: Algorithms for Filtering Data with Humans.




Given a set of data items, we consider the problem of {\em filtering} them based on a set of properties that can be verified by humans. This problem is commonplace in crowdsourcing applications, and yet, to our knowledge, no one has considered the formal optimization of this problem. (Typical solutions use heuristics to solve the problem.) We formally state a few different variants of this problem. We develop deterministic and probabilistic algorithms to optimize the expected cost (i.e., number of questions) and expected error. We experimentally show that our algorithms provide definite gains with respect to other strategies. Our algorithms can be applied in a variety of crowdsourcing scenarios and can form an integral part of any query processor that uses human computation.

Item Type:Article
Uncontrolled Keywords:crowdsourcing, filtering, humans, scoop, optimization, strategies, categorization
ID Code:1011
Deposited By:Aditya Parameswaran
Deposited On:31 Aug 2011 10:03
Last Modified:31 Aug 2011 10:03

Download statistics

Repository Staff Only: item control page