Stanford InfoLab Publication Server

Index Structures for Information Filtering Under the Vector Space Model

Yan, T. and Garcia-Molina, H. (1993) Index Structures for Information Filtering Under the Vector Space Model. Technical Report. Stanford University.




With the ever increasing volumes of information generation, users of information systems are facing an information overload. It is desirable to support information filtering as a complement to traditional retrieval mechanism. The number of users, and thus profiles (representing users' long-term inhandled by an information filtering system is potentially huge, and the system has to process a constant stream of incoming information in a timely fashion. The effciency of the filtering process is thus an important issue. In this paper, we study what data structures and algorithms can be used to effciently perform large-scale information filtering under the vector space model, a retrieval model established as being effective. We apply the idea of the standard inverted index to index user profiles. We devise an alternative to the standard inverted index, in which we, instead of indexing every term in a profile, select only the significant ones to index. We evaluate their performance and show that the indexing methods require orders of magnitude fewer I/Os to process a document than when no index is used. We also show that the proposed alternative performs better in terms of I/O and CPU processing time in many cases.

Item Type:Techreport (Technical Report)
Subjects:Computer Science
Related URLs:Project Homepage
ID Code:18
Deposited By:Import Account
Deposited On:25 Feb 2000 16:00
Last Modified:02 Dec 2008 14:51

Download statistics

Repository Staff Only: item control page