Ramesh, Aditya and Parameswaran, Aditya and Garcia-Molina, Hector and Polyzotis, Neoklis Identifying Reliable Workers Swiftly. Technical Report. Stanford InfoLab.
We study the evaluation and replacement of workers in a crowdsourcing system. We focus on evaluation based on result disagreement among workers, and on policies that replace low accuracy workers with fresh ones from a large pool. We study how long it takes the system to identify an active set of high accuracy workers, and the achieved accuracy of the selected workers. We study through simulations the dynamics of such a system, and we propose a rule of thumb that is helpful for selecting parameters of the evaluation/replacement algorithm.
|Item Type:||Techreport (Technical Report)|
|Uncontrolled Keywords:||quality, crowdsourcing, human computation, spam, replacement|
|Deposited By:||Aditya Parameswaran|
|Deposited On:||06 Jun 2012 17:55|
|Last Modified:||06 Jun 2012 17:55|
Repository Staff Only: item control page