Stanford InfoLab Publication Server

Domain Bias in Web Search

Ieong, Samuel and Mishra, Nina and Sadikov, Eldar and Zhang, Li Domain Bias in Web Search. In: Fifth ACM International Conference on Web Search and Data Mining (WSDM2012), February 8-12, 2012, Seattle, Washingtion, USA.




This paper uncovers a new phenomenon in web search that we call domain bias -- a user’s propensity to believe that a page is more relevant just because it comes from a particular domain. We provide evidence of the existence of domain bias in click activity as well as in human judgments via a comprehensive collection of experiments. We begin by studying the difference between domains that a search engine surfaces and that users click. Surprisingly, we find that despite changes in the overall distribution of surfaced domains, there has not been a comparable shift in the distribution of clicked domains. Users seem to have learned the landscape of the internet and their click behavior has thus become more predictable over time. Next, we run a blind domain test, akin to a Pepsi/Coke taste test, to determine whether domains can shift a user’s opinion of which page is more relevant. We find that domains can actually flip a user’s preference about 25% of the time. Finally, we demonstrate the existence of systematic domain preferences, even after factoring out confounding issues such as position bias and relevance, two factors that have been used extensively in past work to explain user behavior. The existence of domain bias has numerous consequences including, for example, the importance of discounting click activity from reputable domains.

Item Type:Conference or Workshop Item (Paper)
Uncontrolled Keywords:experimentation, measurement, domain bias, web search, ranking, relevance, examination bias, entropy, domain preference graph
ID Code:1021
Deposited By:Eldar Sadikov
Deposited On:12 Dec 2011 00:42
Last Modified:14 Dec 2011 20:59

Download statistics

Repository Staff Only: item control page