Hybrid Strategies for Finding the Max with the Crowd: Technical Report

Abstract

We study the problem of soliciting human crowd workers to find the maximum element in a set of objects, where each object has some unknown intrinsic quality measure. Previous approaches focus on collecting either ratings or pairwise comparisons of items from the crowd, but little study has been done in combining the two crowd interfaces, which we address in this paper. We first examine a common error model for representing pairwise comparisons and extend it to also be appropriate for ratings. We then present and characterize both a maximum-likelihood based approach and a novel PageRank-like approximation for estimating the maximum under this model, given a mixed corpus of evidence containing both ratings and comparisons. Next, we develop heuristic strategies for allocating a crowd question budget between both of these interface types. We show that hybrid approaches utilizing both ratings and comparisons can require substantially fewer questions for the same accuracy as a ratings-only or comparisons-only approach. Finally, we evaluate our algorithms on two sample datasets using Amazon Mechanical Turk, and find that our hybrid approach remains very effective even when using real-world data.