Venetis, Petros and Garcia-Molina, Hector (2012) Dynamic Max Algorithms in Crowdsourcing Environments. Technical Report. Stanford InfoLab.
Our work investigates the problem of retrieving the maximum item from a set in crowdsourcing environments. We focus on tournament algorithms that can for instance select the best Facebook profile that matches a given person or the best photo that describes a given restaurant. Tournament algorithms can be tuned with parameters such as the desired difference of votes between the top-2 voted outcomes, and the maximum number of humans asked to perform a particular task. We propose a strategy for selecting appropriate tournament parameters that attempts to keep monetary cost and latency at a minimum while having quality guarantees. For our experiments, using a human model derived from psychometrics (the Thurstonian model), we provide insights on the effectiveness of our strategy in selecting appropriate tournament parameters and compare our techniques with previous work on tournament tuning.
|Item Type:||Techreport (Technical Report)|
|Uncontrolled Keywords:||crowdsourcing, max algorithms, tournaments, comparisons, uncertainty, dynamic, Thurstonian model|
|Deposited By:||Petros Venetis|
|Deposited On:||07 Aug 2012 17:30|
|Last Modified:||07 Aug 2012 17:32|
Repository Staff Only: item control page