Identifying Reliable Workers Swiftly

Abstract

We study the evaluation and replacement of workers in a crowdsourcing system. We focus on evaluation based on result disagreement among workers, and on policies that replace low accuracy workers with fresh ones from a large pool. We study how long it takes the system to identify an active set of high accuracy workers, and the achieved accuracy of the selected workers. We study through simulations the dynamics of such a system, and we propose a rule of thumb that is helpful for selecting parameters of the evaluation/replacement algorithm.