Stanford InfoLab Publication Server

Stanford WebBase Components and Applications

Cho, Junghoo and Garcia-Molina, Hector and Haveliwala, Taher and Lam, Wang and Paepcke, Andreas and Raghavan, Sriram and Wesley, Gary (2004) Stanford WebBase Components and Applications. Technical Report. Stanford.




We describe the design and performance of WebBase, a tool for Web research. The system includes a highly customizable crawler, a repository for collected Web pages, an indexer for both text and link-related page features, and a high-speed content distribution facility. The distribution module enables researchers world-wide to retrieve pages from WebBase, and stream them across the Internet at high speed. The advantage for the researchers is that they need not all crawl the Web before beginning their research. WebBase has been used by scores of research and teaching organizations world-wide, mostly for investigations into Web topology and linguistic content analysis. After describing the system's architecture, we explain our engineering decisions for each of the WebBase components, and present respective performance measurements.

Item Type:Techreport (Technical Report)
Uncontrolled Keywords:WebBase Web crawler, site crawling; hyperlink indexing; distribution
Subjects:Computer Science > Databases and the Web
Computer Science > Digital Libraries
Projects:Digital Libraries
Related URLs:Project Homepage
ID Code:652
Deposited By:Import Account
Deposited On:08 Jul 2004 17:00
Last Modified:23 Dec 2008 08:45

Download statistics

Repository Staff Only: item control page