Stanford InfoLab Publication Server

Crawler-Friendly Web Servers

Brandman, Onn and Cho, Junghoo and Garcia-Molina, Hector and Shivakumar, Shiva (2000) Crawler-Friendly Web Servers. In: Workshop on Performance and Architecture of Web Servers. (PAWS-2000), June 17-18 , 2000, Santa Clara, CA.




In this paper we study how to make web servers (e.g., Apache) more crawler friendly. Current web servers offer the same interface to crawlers and regular web surfers, even though crawlers and surfers have very different performance requirements. We evaluate simple and easy-to-incorporate modifications to web servers so that there are significant bandwidth savings. Specifically, we propose that web servers export meta-data archives decribing their content.

Item Type:Conference or Workshop Item (Paper)
Uncontrolled Keywords:web servers crawlers robots
Projects:Digital Libraries
Related URLs:Project Homepage
ID Code:446
Deposited By:Import Account
Deposited On:07 Jun 2000 17:00
Last Modified:27 Dec 2008 11:40

Download statistics

Repository Staff Only: item control page