LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correction

Yixuan Wang, Baoxin Wang, Yijun Liu, Dayong Wu, Wanxiang Che

Introduction

Grammatical Error Correction (GEC) for Chinese (CGEC) presents a unique challenge compared to other languages. While identifying errors is difficult, knowing when to refrain from making changes is often harder. Current CGEC models frequently suffer from "over-correction," where they modify grammatically correct phrases into alternative expressions that do not necessarily improve the sentence. This tendency reduces precision and can lead to user frustration.

In this work, researchers from the Harbin Institute of Technology and iFLYTEK Research introduce LM-Combiner, a contextual rewriting model designed to act as a post-processing filter. By decoupling the correction task from the filtering task, LM-Combiner significantly improves system precision without the heavy computational overhead of traditional model ensembles.

The Challenge of Over-Correction

In CGEC, particularly when dealing with native speaker data, errors are often syntactic or semantic rather than simple orthographic mistakes. Models trained on these complex patterns often become over-eager, changing correct segments of text.

To mitigate this, researchers traditionally rely on model ensembles—where multiple models vote on a correction—or perplexity-based (PPL) re-ranking. However, ensembles are computationally expensive, and PPL-based methods often lead to a sharp decline in recall, as the models become too conservative and stop fixing actual errors. LM-Combiner addresses this by treating the problem as a rewriting task rather than a simple ranking or voting task.

Methodology: The LM-Combiner Framework

1. Architecture

LM-Combiner is built upon a causal language model (specifically a GPT-2 architecture). Unlike standard GEC models that map an erroneous input to a corrected output, LM-Combiner takes two inputs: the original sentence and the "candidate" sentence produced by a primary GEC system. By processing both simultaneously, the model can compare the two versions and decide which specific edits to keep, revert, or modify.

2. K-fold Cross Inference

A significant hurdle in training a rewriter is the lack of natural "over-correction" data. To solve this, the authors employed K-fold cross inference. The training data was split into $K$ parts; a GEC model was trained on $K-1$ parts and used to predict the $K$-th part. This process generated a dataset of "natural" errors and over-corrections that a model would realistically produce in a production environment.

3. Gold Labels Merging

To ensure the LM-Combiner focuses specifically on the filtering task, the authors used "Gold Labels Merging." They manually injected missing correct edits from the gold standard into the candidate sentences during training. This forced the model to learn the specific skill of distinguishing between a valid correction and an unnecessary over-correction, rather than trying to learn general correction from scratch.

Experimental Results

Performance on FCGEC

The researchers evaluated LM-Combiner using a BART-Large baseline on the FCGEC benchmark, a fine-grained corpus of native Chinese speaker data. The results demonstrated significant improvements:

  • Precision Boost: The model achieved a precision increase of 18.2 points.
  • Recall Stability: Unlike traditional filtering methods that sacrifice recall for precision, LM-Combiner kept recall virtually unchanged.
  • SOTA Performance: The combined system reached an $F_{0.5}$ score of 51.30, comparable to current State-of-the-Art (SOTA) models on the FCGEC test set.

Enhancing Large Language Models

LM-Combiner also proved effective as a post-processor for black-box Large Language Models (LLMs). When applied to ChatGPT’s zero-shot outputs, LM-Combiner successfully reduced over-corrections and stylistic "hallucinations," demonstrating its utility as a general-purpose enhancement tool that does not require access to the internal weights of the primary model.

Key Insights and Future Directions

The success of LM-Combiner highlights that "filtering" is a fundamentally different task than "correcting." By decoupling these tasks, even a relatively small 62M parameter model can provide substantial gains.

However, limitations remain. The system's overall performance is still bounded by the initial GEC model's recall; if the primary model fails to identify an error, the LM-Combiner cannot fix it. Future research may explore more advanced causal LMs and refined data construction techniques to handle more diverse and subtle error types in Chinese text.

Summary updated

BibTeX
@inproceedings{wang2024lm,
  author = {Yixuan Wang and Baoxin Wang and Yijun Liu and Dayong Wu and Wanxiang Che},
  title = {{LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correction}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=ELe9RXkvAL},
}
Figure