Chinese Spelling Correction as Rephrasing Language Model

Linfeng Liu, Hongqiu Wu, Hai Zhao

Abstract

Traditional Chinese Spelling Correction (CSC) models treat the task as sequence tagging, which often leads to over-conditioning on specific error patterns and poor generalization. This paper introduces ReLM (Rephrasing Language Model), a novel approach that reframes CSC as a rephrasing task. By forcing the model to reconstruct sentences from underlying semantics rather than performing character-to-character mapping, ReLM achieves state-of-the-art performance and superior robustness across diverse domains.

Introduction

Chinese Spelling Correction (CSC) is the essential task of detecting and fixing errors in written Chinese. For years, the dominant paradigm has been sequence tagging, where a model maps an input character directly to a corrected version. However, researchers from Shanghai Jiao Tong University argue that this approach is fundamentally flawed: it allows models to "memorize" specific error patterns without understanding the sentence's global semantics. This makes models brittle in "zero-shot" scenarios and prone to high false-positive rates when encountering unfamiliar contexts.

The Rephrasing Paradigm

To overcome the limitations of tagging, the authors propose shifting the perspective to rephrasing. Humans do not simply swap characters; they grasp the underlying meaning of a clunky or incorrect sentence and then re-express it.

Comparison between tagging-based correction and human-like rephrasing.

As shown in the comparison above, tagging models often rely on seen error patterns (e.g., mapping "age" to "remember"). If the context actually requires a different word, the tagging model fails. In contrast, a rephrasing approach focuses on the semantics—what the sentence is trying to convey—to generate the correct output.

Methodology: ReLM

The proposed ReLM (Rephrasing Language Model) implements this paradigm using a BERT-based architecture with a specific input template and training objectives.

1. Input Template and Objective

Instead of standard input-output pairs, ReLM concatenates the source sentence ($X$) with a sequence of mask tokens ($M$) of the same length: [Source Sentence] <sep> [Mask][Mask]...[Mask]. The model is then trained to infill these masks to produce the corrected sentence. This structure forces the model to encode the source into a semantic space before generating the output.

2. Multi-Task Prompting

ReLM is designed to be a versatile language representation model. By incorporating trainable prompt characters, the model can perform CSC alongside other tasks like sentiment analysis or natural language inference (NLI) without the performance degradation typical of specialized tagging models.

ReLM architecture showing the rephrasing template and multi-task prompting capabilities.

3. Auxiliary Masked Language Modeling

To prevent the model from learning a simple "copying" shortcut, the authors use auxiliary masking on non-error characters during training. This ensures the model relies on linguistic context to restore the entire sentence rather than just focusing on suspected error positions.

Experimental Results

ReLM was evaluated against several rigorous benchmarks, including ECSpell (specialized domains) and LEMON (real-life errors).

  • Domain Generalization: In the specialized LAW domain of ECSpell, ReLM achieved an F1 score of 91.2, a massive improvement over the standard BERT-tagging baseline (39.8).
  • Zero-Shot Performance: On the LEMON dataset, ReLM improved the average F1 score from 49.7 (previous state-of-the-art) to 51.2, demonstrating superior generalization to unseen data.
  • Reliability: In the Official Documents (ODW) domain, ReLM slashed the False Positive Rate (FPR) to 2.2, compared to 13.9 for standard tagging models. This reduction in "over-correction" makes ReLM significantly more viable for real-world applications.

Conclusion

By reimagining Chinese Spelling Correction as a rephrasing task, ReLM moves beyond the brittle pattern-matching of traditional tagging models. Its ability to prioritize semantics leads to more human-like corrections, better performance in specialized domains, and a more robust foundation for multi-task natural language processing.

Summary updated

BibTeX
@inproceedings{liu2024chinese,
  author = {Linfeng Liu and Hongqiu Wu and Hai Zhao},
  title = {{Chinese Spelling Correction as Rephrasing Language Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29829},
}
Figure