Prasad, Jyotika and Paepcke, Andreas (2008) CoreEx: Content Extraction from Online News Articles. Technical Report. Stanford InfoLab.
We developed and tested a heuristic technique for extracting the main article from news site Web pages. We construct the DOM tree of the page and score every node based on the amount of text and the number of links it contains. The method is site-independent and does not use any language- based features. We tested our algorithm on a set of 1120 news article pages from 27 domains. This dataset was also used elsewhere to test the performance of another, state-of- the-art baseline system. Our algorithm achieved over 97% precision and 98% recall, and an average processing speed of under 15ms per page. This precision/recall performance is slightly below the baseline system, but our approach requires significantly less computational work.
|Item Type:||Techreport (Technical Report)|
|Uncontrolled Keywords:||content extraction, web sociologists workbench|
|Subjects:||Computer Science > Digital Libraries|
|Deposited By:||Import Account|
|Deposited On:||02 Jun 2008 17:00|
|Last Modified:||10 Dec 2008 16:18|
Repository Staff Only: item control page