Abstract
Traditional Chinese Spelling Correction (CSC) models treat the task as sequence tagging, which often leads to over-conditioning on specific error patterns and poor generalization. This paper introduces ReLM (Rephrasing Language Model), a novel approach that reframes CSC as a rephrasing task. By forcing the model to reconstruct sentences from underlying semantics rather than performing character-to-character mapping, ReLM achieves state-of-the-art performance and superior robustness across diverse domains.
Introduction
Chinese Spelling Correction (CSC) is the essential task of detecting and fixing errors in written Chinese. For years, the dominant paradigm has been sequence tagging, where a model maps an input character directly to a corrected version. However, researchers from Shanghai Jiao Tong University argue that this approach is fundamentally flawed: it allows models to "memorize" specific error patterns without understanding the sentence's global semantics. This makes models brittle in "zero-shot" scenarios and prone to high false-positive rates when encountering unfamiliar contexts.
The Rephrasing Paradigm
To overcome the limitations of tagging, the authors propose shifting the perspective to rephrasing. Humans do not simply swap characters; they grasp the underlying meaning of a clunky or incorrect sentence and then re-express it.
As shown in the comparison above, tagging models often rely on seen error patterns (e.g., mapping "age" to "remember"). If the context actually requires a different word, the tagging model fails. In contrast, a rephrasing approach focuses on the semantics—what the sentence is trying to convey—to generate the correct output.
Methodology: ReLM
The proposed ReLM (Rephrasing Language Model) implements this paradigm using a BERT-based architecture with a specific input template and training objectives.
1. Input Template and Objective
Instead of standard input-output pairs, ReLM concatenates the source sentence ($X$) with a sequence of mask tokens ($M$) of the same length: [Source Sentence] <sep> [Mask][Mask]...[Mask]. The model is then trained to infill these masks to produce the corrected sentence. This structure forces the model to encode the source into a semantic space before generating the output.
2. Multi-Task Prompting
ReLM is designed to be a versatile language representation model. By incorporating trainable prompt characters, the model can perform CSC alongside other tasks like sentiment analysis or natural language inference (NLI) without the performance degradation typical of specialized tagging models.
3. Auxiliary Masked Language Modeling
To prevent the model from learning a simple "copying" shortcut, the authors use auxiliary masking on non-error characters during training. This ensures the model relies on linguistic context to restore the entire sentence rather than just focusing on suspected error positions.
Experimental Results
ReLM was evaluated against several rigorous benchmarks, including ECSpell (specialized domains) and LEMON (real-life errors).
- Domain Generalization: In the specialized LAW domain of ECSpell, ReLM achieved an F1 score of 91.2, a massive improvement over the standard BERT-tagging baseline (39.8).
- Zero-Shot Performance: On the LEMON dataset, ReLM improved the average F1 score from 49.7 (previous state-of-the-art) to 51.2, demonstrating superior generalization to unseen data.
- Reliability: In the Official Documents (ODW) domain, ReLM slashed the False Positive Rate (FPR) to 2.2, compared to 13.9 for standard tagging models. This reduction in "over-correction" makes ReLM significantly more viable for real-world applications.
Conclusion
By reimagining Chinese Spelling Correction as a rephrasing task, ReLM moves beyond the brittle pattern-matching of traditional tagging models. Its ability to prioritize semantics leads to more human-like corrections, better performance in specialized domains, and a more robust foundation for multi-task natural language processing.
Summary updated