Stanford InfoLab Publication Server

CoreEx: Content Extraction from Online News Articles

Prasad, Jyotika and Paepcke, Andreas (2008) CoreEx: Content Extraction from Online News Articles. Technical Report. Stanford InfoLab.




We developed and tested a heuristic technique for extracting the main article from news site Web pages. We construct the DOM tree of the page and score every node based on the amount of text and the number of links it contains. The method is site-independent and does not use any language- based features. We tested our algorithm on a set of 1120 news article pages from 27 domains. This dataset was also used elsewhere to test the performance of another, state-of- the-art baseline system. Our algorithm achieved over 97% precision and 98% recall, and an average processing speed of under 15ms per page. This precision/recall performance is slightly below the baseline system, but our approach requires significantly less computational work.

Item Type:Techreport (Technical Report)
Uncontrolled Keywords:content extraction, web sociologists workbench
Subjects:Computer Science > Digital Libraries
ID Code:832
Deposited By:Import Account
Deposited On:02 Jun 2008 17:00
Last Modified:10 Dec 2008 16:18

Download statistics

Repository Staff Only: item control page