Abstract
Manually annotated datasets are essential for training high-performance NLP models, but they are expensive to produce. A significant challenge arises when multiple corpora for the same task use incompatible annotation standards, preventing researchers from easily combining them. This paper presents a stacking-based method to automatically adapt knowledge from a large source corpus to a smaller target corpus with a different annotation style. Using Chinese word segmentation and Part-of-Speech (POS) tagging as a case study, the authors demonstrate that this approach significantly reduces error rates and improves downstream parsing accuracy.
The Challenge of Incompatible Standards
Supervised NLP models rely heavily on manually annotated data. However, linguistic theories often differ, leading research groups to create datasets for the same task using conflicting guidelines. This is particularly problematic in Chinese NLP, where the lack of spaces between words and minimal morphology makes the definition of a "word" highly subjective.
For instance, the Penn Chinese Treebank (CTB) and the People's Daily (PD) corpus often disagree on word boundaries and POS tags. While domain adaptation (adapting across genres) is well-studied, annotation-style adaptation—where the labeling rules themselves conflict—remains a difficult hurdle for leveraging large-scale data.
Stacking-Based Annotation Adaptation
The authors treat Chinese word segmentation and POS tagging as a character-level classification problem, assigning tags for word position (Begin, Middle, End, Single) and POS. To bridge the gap between standards, they implement a "stacking" strategy that allows a target model to learn from a source model's predictions.
The process involves three main steps:
- Source Classifier Training: A model is trained on the large source corpus (PD) to capture its specific linguistic knowledge.
- Feature Generation: This source classifier is run over the raw text of the target corpus (CTB), producing predictions according to the source standard.
- Target Classifier Training: A new model is trained on the target corpus. Crucially, its feature set is augmented with the predictions from Step 2. These "guide features" allow the target model to automatically learn the transformation rules required to map the source standard to the target standard.
Experimental Results
The study utilized the massive People's Daily (PD) corpus (5.86 million words) as the source and the smaller, but syntactically rich, Penn Chinese Treebank (CTB) 5.0 (0.47 million words) as the target. The adaptation strategy yielded significant improvements:
- Word Segmentation: The F1 score rose from 97.35% to 98.15%, a 30.2% error reduction.
- Joint Segmentation & POS Tagging: The F1 score improved from 93.06% to 94.03%, a 14% error reduction.
- Downstream Impact: When the improved segmentation was used for syntactic parsing, the parsing F1 score increased from 80.28% to 81.07%, effectively reducing the error propagation from segmentation to parsing by 38%.
Conclusion
This stacking approach provides a simple yet effective way to integrate knowledge from corpora with conflicting annotation standards. By treating the source model's output as a feature rather than a ground truth, the system implicitly learns to "translate" between linguistic standards. Future work may explore using n-best lists or lattices as guide features to further improve error tolerance and applying this method to more complex structures like dependency treebanks.
Summary updated