Introduction
In the modern era of information overload, Novelty Mining (NM) has emerged as a critical technology for filtering chronological text streams. The goal is to identify documents or sentences that provide information previously unseen by the user. While novelty mining is well-established for English, applying these techniques to Chinese presents unique linguistic challenges, primarily due to the lack of natural word boundaries.
The Challenge of Chinese Text
Unlike English, which uses spaces to demarcate words, Chinese text is a continuous stream of characters. Consequently, any automated analysis must begin with "word segmentation." This process is inherently prone to errors, which can propagate through the system. Furthermore, English-centric techniques like stemming (e.g., reducing "running" to "run") are inapplicable to Chinese. The research presented here investigates how preprocessing choices—specifically segmentation and Part-of-Speech (POS) tagging—influence the efficacy of novelty detection in Chinese compared to English.
Methodology
The researchers proposed a three-stage pipeline: Preprocessing, Categorization, and Novelty Mining. The innovation lies in the Chinese-specific preprocessing stage.
Preprocessing and POS Filtering
After using the ICTCLAS tool for initial segmentation, the authors tested two distinct rules for filtering the resulting tokens:
- Rule 1 (Loose Filtering): Only non-meaningful words such as pronouns, prepositions, and punctuation are removed.
- Rule 2 (Strict Filtering): Only core content words—nouns (including proper names and locations), verbs, adjectives, and adverbs—are retained.
The similarity between a new document and previous entries in its topic "bin" is calculated using Cosine Similarity with Term Frequency (TF) weighting. A document is flagged as "novel" if its similarity score falls below a predefined threshold.
Experimental Results
The system was evaluated using two benchmarks: the APWSJ (Associated Press & Wall Street Journal) dataset for document-level mining and the TREC 2004 Novelty Track for sentence-level mining.
The Superiority of Strict Filtering
The most significant finding was that Rule 2 consistently outperformed Rule 1 across both document and sentence-level tests. By focusing exclusively on high-semantic-weight words, the system effectively ignores the "noise" or "junk tokens" often created by segmentation errors. In Chinese, where function words do not undergo inflection, they act more as noise than useful features for novelty detection.
Sentence vs. Document Sensitivity
The study revealed that sentence-level novelty mining is remarkably resilient to preprocessing imperfections. Even with imperfect segmentation, performance at the sentence level nearly matched English systems. In contrast, document-level mining proved much more sensitive to the quality of the "bag of words" generated during preprocessing. The authors attribute the document-level performance gap with English to the inherent difficulty of obtaining a clean token set in Chinese.
Conclusion
The research demonstrates that for Chinese novelty mining, "less is more." Strict POS filtering acts as a powerful noise-reduction mechanism, mitigating the impact of segmentation errors. While a performance gap remains between Chinese and English systems at the document level, the proposed strict filtering approach nearly closes this gap for sentence-level tasks. Future work will focus on more complex word combinations and real-time application for live news feeds.
Summary updated