Combining a segmentation-like approach and a density-based approach in content extraction

Shuang Lin, Jie Chen, Zhendong NiuTsinghua 2012 · DOI

Abstract

Density-based approaches in content extraction, whose task is to extract contents from Web pages, are commonly used to obtain page contents that are critical to many Web mining applications. However, traditional density-based approaches cannot effectively manage pages that contain short contents and long noises. To overcome this problem, in this paper, we propose a content extraction approach for obtaining content from news pages that combines a segmentation-like approach and a density-based approach. A tool called BlockExtractor was developed based on this approach. BlockExtractor identifies contents in three steps. First, it looks for all Block-Level Elements (BLE) & Inline Elements (IE) blocks, which are designed to roughly segment pages into blocks. Second, it computes the densities of each BLE&IE block and its element to eliminate noises. Third, it removes all redundant BLE&IE blocks that have emerged in other pages from the same site. Compared with three other density-based approaches, our approach shows significant advantages in both precision and recall.
BibTeX
@inproceedings{lin2012combining,
  author = {Shuang Lin and Jie Chen and Zhendong Niu},
  title = {{Combining a segmentation-like approach and a density-based approach in content extraction}},
  booktitle = {Tsinghua},
  year = {2012},
  doi = {10.1109/tst.2012.6216755},
}

Loading related content...

Figure