Abstract
Chinese Word Segmentation (CWS) is a fundamental task in Chinese natural language processing, yet it remains challenged by Out-of-Vocabulary (OOV) words. This paper presents WeiDC (Weighted self Distillation for Chinese word segmentation), a framework that leverages self-distillation and a novel weighting mechanism to improve segmentation accuracy. By focusing on the "information gap" between predicted and ground-truth labels, WeiDC allows models to better capture rare and unseen words without the need for external dictionaries or complex features. Experimental results on SIGHAN benchmarks demonstrate that WeiDC achieves state-of-the-art performance, particularly in OOV recall.
Introduction: The OOV Challenge in CWS
Unlike English, Chinese text lacks explicit word boundaries, making Chinese Word Segmentation (CWS) the essential first step for downstream NLP tasks. While Transformer-based models like BERT have significantly advanced the field, they struggle with Out-of-Vocabulary (OOV) words—terms not encountered during training. Research indicates that OOV words contribute to five times more errors than simple segmentation ambiguities.
Traditional approaches to mitigate this issue often rely on external resources, such as pre-defined dictionaries, n-gram features, or multi-criteria datasets. However, these methods increase complexity and depend on the quality of external data. WeiDC addresses this by asking whether a model can improve its own robustness through self-teaching.
The WeiDC Framework
The WeiDC architecture follows a standard sequence labeling pipeline but introduces a specialized training regime. It consists of an Embedding layer, a Transformer-based Encoder (such as BERT or RoBERTa), and a Softmax Decoder.
The core innovation lies in two components:
- Self-Distillation: Instead of a traditional teacher-student setup involving two different models, WeiDC uses the best-performing historical version of the model itself as the "Teacher." This teacher provides pseudo-labels to guide the current "Student" iteration.
- Weighting Mechanism: To prioritize difficult samples, the authors developed four weighting modules ($w^1$ to $w^4$). These modules calculate an "information gap" based on the alignment (or lack thereof) between the teacher's predictions, the student's predictions, and the true labels. If the models struggle with a particular character, the weight is increased, forcing the student to focus on these high-value, often rare, tokens.
Experimental Setup
The framework was evaluated on the SIGHAN Bakeoff benchmark, utilizing four standard datasets: MSR (Microsoft Research), PKU (Peking University), AS (Academia Sinica), and CITYU (City University of Hong Kong). These datasets vary in their OOV rates and domain coverage, providing a comprehensive test of generalization. Additionally, the method was tested on Named Entity Recognition (NER) tasks (WEIBO, RESUME, MSRA) to verify its broader applicability to sequence labeling.
Key Results and Insights
WeiDC demonstrates significant improvements over baseline models and existing state-of-the-art (SOTA) methods.
As shown in the results table, WeiDC (using RoBERTa) achieves an average F-score of 97.43% and an average OOV recall ($R_{OOV}$) of 85.96%. Notably, it outperforms models that utilize external lexicons or complex n-gram features.
Key Findings:
- OOV Breakthrough: The weighted distillation approach specifically targets the "hard" cases, leading to a 1.71% average improvement in OOV recall compared to standard BERT.
- Teacher Selection: Using the "best historical model" ($D_{best}$) as a teacher proved more effective than using the model from the immediate previous iteration ($D_{last}$), as it provides more stable and high-quality guidance.
- Efficiency: The success of a simple Softmax decoder suggests that the quality of the learned representations via distillation is more critical than the complexity of the decoding algorithm (like CRF).
- Low-Resource Robustness: In experiments where training data was restricted to 5%, WeiDC maintained significantly higher OOV recall than standard models, highlighting its utility in data-scarce environments.
Conclusion and Future Work
WeiDC offers a streamlined, effective method for improving CWS by making models "teach themselves" to handle difficult words. While it achieves SOTA results, limitations remain, such as handling ambiguous adjacent words that have never been seen together. Future research may explore automated weighting strategies to replace the current hand-crafted modules and extend the framework to non-sequential NLP tasks.
Summary updated