Dynamic Max Algorithms in Crowdsourcing Environments

Abstract

Our work investigates the problem of retrieving the maximum item from a set in crowdsourcing environments. We focus on tournament algorithms that can for instance select the best Facebook profile that matches a given person or the best photo that describes a given restaurant. Tournament algorithms can be tuned with parameters such as the desired difference of votes between the top-2 voted outcomes, and the maximum number of humans asked to perform a particular task. We propose a strategy for selecting appropriate tournament parameters that attempts to keep monetary cost and latency at a minimum while having quality guarantees. For our experiments, using a human model derived from psychometrics (the Thurstonian model), we provide insights on the effectiveness of our strategy in selecting appropriate tournament parameters and compare our techniques with previous work on tournament tuning.