Cho, Junghoo and Garcia-Molina, Hector and Haveliwala, Taher and Lam, Wang and Paepcke, Andreas and Raghavan, Sriram and Wesley, Gary (2004) Stanford WebBase Components and Applications. Technical Report. Stanford.
We describe the design and performance of WebBase, a tool for Web research. The system includes a highly customizable crawler, a repository for collected Web pages, an indexer for both text and link-related page features, and a high-speed content distribution facility. The distribution module enables researchers world-wide to retrieve pages from WebBase, and stream them across the Internet at high speed. The advantage for the researchers is that they need not all crawl the Web before beginning their research. WebBase has been used by scores of research and teaching organizations world-wide, mostly for investigations into Web topology and linguistic content analysis. After describing the system's architecture, we explain our engineering decisions for each of the WebBase components, and present respective performance measurements.
|Item Type:||Techreport (Technical Report)|
|Uncontrolled Keywords:||WebBase Web crawler, site crawling; hyperlink indexing; distribution|
|Subjects:||Computer Science > Databases and the Web|
Computer Science > Digital Libraries
|Related URLs:||Project Homepage||http://www-diglib.stanford.edu/diglib/pub/|
|Deposited By:||Import Account|
|Deposited On:||08 Jul 2004 17:00|
|Last Modified:||23 Dec 2008 08:45|
Repository Staff Only: item control page