Machine learning methods for Chinese web page categorization

Ji He, Ah‐Hwee Tan, Chew‐Lim Tan2000 · DOI

TL;DR

This paper evaluates three machine learning methods (kNN, SVM, ARAM) for classifying Chinese web pages, finding that ARAM and kNN slightly outperform SVM, especially for smaller categories. Notably, incorporating domain knowledge as rules into ARAM further boosted its performance on well-defined small categories, suggesting a promising avenue for improving text classification with limited data.

What Problem Does This Paper Address?

Automatically categorizing text documents is a well-established problem, with numerous methods explored for English. However, applying these techniques to Chinese presents unique challenges. Unlike English, Chinese lacks natural word delimiters, making word segmentation a critical and imperfect preprocessing step that introduces inherent errors. Furthermore, there's a scarcity of publicly available Chinese corpora for robust evaluation. This paper aims to benchmark the performance of three prominent machine learning classifiers – k Nearest Neighbor (kNN), Support Vector Machines (SVM), and Adaptive Resonance Associative Map (ARAM) – on the challenging task of Chinese web page categorization, using a newly created corpus.

Methods

The authors compare three statistical machine learning methods:

  • k Nearest Neighbor (kNN): A "lazy" learning algorithm that classifies new data points based on the majority class of their k nearest neighbors in the training set. It doesn't perform explicit offline learning but rather on-line scoring.
  • Support Vector Machines (SVM): A method that finds an optimal separating hyperplane to distinguish between classes. It's known for its computational efficiency during classification, especially for large datasets, by relying on "support vectors." The paper uses the SVMlight implementation.
  • Adaptive Resonance Associative Map (ARAM): A predictive self-organizing neural network that performs incremental supervised learning. It uses a fuzzy ART architecture and can incorporate domain knowledge in the form of IF-THEN rules. A key aspect is its ability to abstract patterns and potentially handle noisy data.

The paper also details their approach to feature selection and extraction:

  • Word Segmentation: A word-class bi-gram model is used, acknowledging that perfect precision is not achievable but aiming for reasonably low error rates.
  • Keyword Selection: The $\chi^2$ (CHI) statistic is employed as a ranking metric, chosen for its proven effectiveness over other measures like term strength and document frequency.
  • Feature Vector Creation: Documents are converted into frequency vectors, then weighted using inverse document frequency (IDF) and L1-normalization to produce the final keyword feature vector.

Datasets

The primary dataset used is a Chinese web corpus collected in-house. For their experiments, they used a subset consisting of 8 top-level categories and over 6,000 documents. The corpus was further processed for binary classification experiments, where each category was treated as positive against the other seven as negative. The corpus was partitioned into training and testing sets, with the training set being at least 2.5 times the size of the testing set.

The paper also mentions a Chinese news article corpus from People's Daily used in prior work (He et al., 2000), but the current experiments focus on the more challenging web page data.

Key Insights and Ideas

  • ARAM's Strength in Small Categories: The most striking finding is ARAM's superior performance (macro-averaged F1 score) on smaller categories compared to SVM. This suggests that its pattern abstraction and category learning mechanisms are more robust when training data is scarce.
  • Domain Knowledge Boosts ARAM: The ability to insert domain knowledge as IF-THEN rules into ARAM significantly improved its performance, particularly for small, well-defined categories. This highlights the value of combining statistical learning with explicit knowledge representation.
  • kNN's Stable Performance: kNN demonstrated a relatively stable performance across both large and small categories, likely due to its "lazy" learning approach which doesn't rely on explicit offline learning of category representations.
  • SVM's Challenge with Sparse Data: SVM's performance was generally lower on small categories. This is attributed to the potential for the optimal separating hyperplane (OSH) to be less representative when training data is sparse and margins are narrow.
  • Web Page Data Complexity: Classifying real-world web pages is a "much more challenging task" than news articles due to the diversity in document length, style, and content.

Key Results

  • Micro-averaged scores: The three classifiers (kNN, SVM, ARAM) showed roughly comparable micro-averaged performance, largely influenced by their performance on large categories (Biz, IT, Joy). kNN was marginally better in this regard.
  • Macro-averaged scores: ARAM significantly outperformed kNN and SVM in macro-averaged F1 scores, indicating its better ability to handle smaller categories.
  • ARAM with Rules: Inserting rules into ARAM showed a noticeable improvement, especially for categories with fewer training patterns.
  • Category-Specific Performance:
    • Art, Belief, Edu, Sci (smaller categories) showed more pronounced differences in macro-averaged performance, with ARAM and kNN generally outperforming SVM.
  • Biz, IT, Joy (larger categories) showed more comparable performance across classifiers.

Limitations and Future Work

  • Word Segmentation Imperfection: The inherent errors from Chinese word segmentation remain a persistent challenge that affects all classifiers.
  • ARAM Rule Insertion Impact: While rule insertion improved ARAM, the impact was not universally dramatic, suggesting that the effectiveness depends on the quality and relevance of the abstractable domain knowledge.
  • Future Work: The authors suggest that ARAM's scalability and ability to abstract patterns make it more suitable for handling very large datasets. They also emphasize the potential for domain knowledge to complement limited training data, especially for well-defined categories, hinting at further exploration of hybrid approaches.

Summary updated

Abstract

This paper reports our evaluation of k Nearest Neighbor (kNN), Support Vector Machines (SVM), and Adaptive Resonance Associative Map (ARAM) on Chinese web page classification. Benchmark experiments based on a Chinese web corpus showed that their predictive performance were roughly comparable although ARAM and kNN slightly outperformed SVM in small categories. In addition, inserting rules into ARAM helped to improve performance, especially for small well-defined categories.
BibTeX
@misc{he2000machine,
  author = {Ji He and Ah‐Hwee Tan and Chew‐Lim Tan},
  title = {{Machine learning methods for Chinese web page categorization}},
  year = {2000},
  doi = {10.3115/1117769.1117785},
}
Figure