Stanford InfoLab Publication Server

On Indexing Error-Tolerant Set Containment

Agrawal, Parag and Arasu, Arvind and Kaushik, Raghav (2010) On Indexing Error-Tolerant Set Containment. In: SIGMOD.




Prior work has identified set based comparisons as a useful primitive for supporting a wide variety of similarity functions in record matching. Accordingly, various techniques have been proposed to improve the performance of set similarity lookups. However, this body of work focuses almost exclusively on symmetric notions of set similarity. In this paper, we study the indexing problem for the asymmetric Jaccard containment similarity function that is an error-tolerant variation of set containment. We enhance this similarity function to also account for string transformations that reflect synonyms such as “Bob” and “Robert” referring to the same first name. We propose an index structure that builds inverted lists on carefully chosen token-sets and a lookup algorithm using our index that is sensitive to the output size of the query. Our experiments over real life data sets show the benefits of our techniques. To our knowledge, this is the first paper that studies the indexing problem for Jaccard containment in the presence of string transformations.

Item Type:Conference or Workshop Item (Paper)
ID Code:971
Deposited By:Parag Agrawal
Deposited On:11 Jun 2010 05:10
Last Modified:11 Jun 2010 05:10

Download statistics

Repository Staff Only: item control page