Towards combining web classification and web information extraction: a case study

Ping Luo, Fen Lin, Yuhong Xiong, Yong Zhao, Zhongzhi ShiKDD 2009 · DOI

TL;DR

This paper tackles the inefficiency of traditional, separate approaches to web classification and information extraction by proposing a unified Conditional Random Field (CRF) model. This integrated approach simultaneously identifies relevant web pages and extracts their metadata, leading to significant improvements in both tasks.

What Problem Does This Paper Address?

The web is a vast repository of information, and extracting structured data from it is crucial for sophisticated applications like vertical search engines. Traditionally, this process involves two distinct steps: first, classifying web pages to identify those relevant to a specific domain (e.g., online course pages), and second, extracting specific pieces of information (metadata) from these identified pages (e.g., course title, ID, time).

The problem with this decoupled approach is that errors in the first step (classification) propagate to the second step (extraction), leading to accumulated errors and suboptimal performance. For instance, if a page is misclassified as a course page, extraction attempts will be made on irrelevant content. Conversely, if a genuine course page is missed by the classifier, its metadata will never be extracted. This paper argues that these two tasks are not independent but rather mutually dependent, and proposes a method to leverage this dependency for better results.

Methods

The core contribution of this paper is the proposal of a unified probabilistic graphical model based on Conditional Random Fields (CRFs) to jointly perform web classification and web information extraction.

  • Unified Model: Instead of two separate models, a single CRF model is used. This model treats the web page itself as one variable (for classification) and the DOM nodes within the page as other variables (for information extraction).
  • Simultaneous Optimization: The CRF allows for the simultaneous prediction of both the page's class and the metadata labels for its DOM nodes.
  • Leveraging Dependencies: The model explicitly incorporates the dependencies between the page classification and the information extraction tasks. This includes both "forward" dependencies (classification helps extraction) and "backward" dependencies (extracted metadata can help refine classification).
  • Conditional Random Fields (CRFs): CRFs are a type of undirected graphical model that are well-suited for structured prediction tasks. They model the conditional probability of a set of output labels given a set of observations.
  • Constrained Output: A key innovation is the reduction of the output label space by incorporating domain-specific constraints. For example, the paper assumes that a course homepage must contain a course title, and a non-course homepage will not. This constraint significantly simplifies the model and enables efficient, exact inference.
  • Feature Engineering: The paper details specific features used for both classification (e.g., course-related keywords, their HTML formatting and position) and extraction (e.g., font size, color, position, and content-based features for course titles).

Datasets

The experiments were conducted using a dataset collected for the "OfCourse" project, which is an online course search engine.

  • Training and Evaluation Data: The dataset consists of 530 course homepages and 1200 non-course pages. For course homepages, the course title, ID, and time were manually annotated.
  • Benchmark: The paper introduces a benchmark for evaluating the joint task of course homepage identification and course title extraction.

Key Insights and Ideas

  • Interdependence is Key: The most significant insight is that web classification and information extraction are not independent tasks. Treating them separately leads to error propagation, while a joint model can achieve mutual enhancement.
  • Backward Dependency Matters: The paper highlights the importance of the "backward" dependency – how information extracted from a page can inform its classification. This is demonstrated with examples where the presence or absence of a course title helps correctly classify a page.
  • Constraints Simplify Complexity: By incorporating domain-specific constraints (e.g., a course page must have a title), the authors significantly reduce the search space for the CRF model, enabling efficient and exact inference. This is a crucial practical insight for applying complex graphical models.
  • Unified Framework for Diverse Tasks: The proposed CRF framework is presented as a general approach that can be adapted to combine various web analysis tasks, not just course information extraction.

Key Results

The proposed joint optimization model significantly outperformed the baseline methods in both web classification and web information extraction.

  • Course Homepage Identification: The joint model achieved an F1 score of 0.898, compared to 0.858 for the best baseline (Local training and Separate Inference).
  • Course Title Extraction: The joint model achieved an F1 score of 0.842, compared to 0.811 for the best baseline.
  • Statistical Significance: The improvements were shown to be statistically significant using a 95% confidence interval.
  • Application-Dependent Features: The inclusion of application-dependent features for course title extraction led to an average improvement of 8.72% in accuracy.

Limitations and Future Work

  • Assumptions: The model relies on certain assumptions, such as a course homepage containing a course title and that course titles are generally prominent. The authors acknowledge that rare exceptions exist.
  • DOM Node Splitting: The paper mentions the need for preprocessing to split DOM nodes that contain multiple pieces of metadata (e.g., a single node containing course title, ID, and time). This preprocessing step is not detailed extensively.
  • Generalizability: While the methodology is presented as generalizable, the specific features and constraints are tailored to the course search domain. Adapting it to other domains would require re-engineering these aspects.
  • Future Work: The authors suggest that this approach can be generalized to other web applications, such as classifying news or blog pages and extracting their respective metadata. The paper also hints at further theoretical analysis of the relationship between regularization parameters and performance metrics.

Summary updated

BibTeX
@inproceedings{luo2009towards,
  author = {Ping Luo and Fen Lin and Yuhong Xiong and Yong Zhao and Zhongzhi Shi},
  title = {{Towards combining web classification and web information extraction: a case study}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2009},
  doi = {10.1145/1557019.1557152},
}
Figure