Chinese Spelling Correction as Rephrasing Language Model

Linfeng Liu, Hongqiu Wu, Hai Zhao

Abstract

Traditional Chinese Spelling Correction (CSC) has long relied on sequence tagging, where models learn to map erroneous characters to their correct counterparts. However, this paradigm often results in models that are excessively conditioned on specific error patterns, leading to poor generalization and high false positive rates. We present ReLM (Rephrasing Language Model), a novel approach that reframes CSC as a sentence-level rephrasing task. By utilizing a non-autoregressive infilling objective, ReLM forces the model to capture global semantics before reconstructing the sentence. Our results across multiple benchmarks, including ECSpell and LEMON, demonstrate that ReLM significantly outperforms existing tagging-based methods, particularly in zero-shot and multi-task scenarios.

Introduction: The Limitations of Sequence Tagging

The standard approach to CSC involves character-level sequence tagging, typically using pre-trained models like BERT. While effective for common errors, this method suffers from a "critical flaw": models become overly reliant on memorized error-correction pairs. For instance, if a model frequently corrects "age" (a common typo for "remember") to "remember," it may fail to recognize when "age" is actually a typo for "not" or when "age" is used correctly. This lack of semantic understanding leads to high over-correction rates and limited transferability to other NLP tasks.

Comparison between traditional tagging-based correction (a) and the proposed human-centric rephrasing approach (b).

ReLM: A Rephrasing Paradigm

ReLM shifts the training objective from 1-to-1 tagging to a rephrasing-by-infilling paradigm. Given an input sentence $X$, the model is tasked with filling a sequence of mask tokens $M$ of equal length. This architecture forces the encoder to map the entire source sentence into a semantic space before the decoder reconstructs the correct version.

To ensure the model learns robust language patterns rather than simple character alignment, ReLM employs an auxiliary Masked Language Modeling (MLM) objective, randomly masking non-error characters during training. Unlike autoregressive models like GPT, ReLM uses an encoder-only architecture to infill all slots simultaneously, maintaining the fixed-length constraints typical of CSC tasks while improving accuracy through global context.

Experimental Results and Insights

ReLM demonstrates substantial improvements over state-of-the-art (SotA) tagging models across several rigorous benchmarks:

  • ECSpell Performance: In the specialized "LAW" domain, ReLM achieved an F1 score of 91.2, a significant jump from the 76.1 achieved by previous BERT-based tagging models.
  • Zero-Shot Superiority: On the LEMON dataset, ReLM improved the average F1 from 49.7 to 51.2, highlighting its superior generalization to unseen error types.
  • Multi-Task Stability: While tagging models often see performance degradation (up to 32%) when trained on multiple tasks (e.g., CSC combined with news classification via TNEWS), ReLM's performance remains stable or even improves, benefiting from the complementary semantic signals of diverse tasks through prompt tuning.

Conclusion and Future Work

By treating Chinese Spelling Correction as a rephrasing task, ReLM overcomes the limitations of local error-pattern memorization and mimics human-centric logic. While the model currently assumes fixed sentence lengths and requires further domain-specific tuning for highly specialized fields (such as complex judicial logic), it represents a significant shift toward more semantic-driven text correction. Future work may involve relaxing the fixed-length constraint to handle insertion and deletion errors more effectively.

Summary updated

BibTeX
@inproceedings{liu2024chinese,
  author = {Linfeng Liu and Hongqiu Wu and Hai Zhao},
  title = {{Chinese Spelling Correction as Rephrasing Language Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29829},
}
Figure