Abstract
This paper improves tagging-based Grammatical Error Correction (GEC) models by addressing two major bottlenecks: extreme label imbalance and the "entanglement" of error detection and correction tasks. By implementing Focal Loss to prioritize hard-to-fix errors and a self-attention matching module to decouple the tagging heads, the authors achieved significant performance gains across three major Chinese GEC benchmarks: MuCGEC, FCGEC, and MCSCSet.
Introduction
Grammatical Error Correction (GEC) is typically handled in one of two ways: Sequence-to-Sequence (Seq2Seq) models, which "rewrite" the sentence, and Tagging-based models (like the popular GECToR), which predict edit operations (KEEP, DELETE, REPLACE, etc.) for each token. While tagging-based models are much faster and more efficient, they face two specific hurdles:
- Label Imbalance: In a typical sentence, most words are correct. This means the "KEEP" tag dominates the dataset. Standard cross-entropy loss allows the model to achieve high accuracy just by predicting "KEEP" everywhere, making it "lazy" when it comes to actually identifying and fixing rare errors.
- Tagging Entanglement: Most models use the same hidden representation (from a Transformer encoder) for two different tasks: detecting if there is an error and deciding how to fix it. The authors argue that these tasks interfere with each other when forced to share the exact same linear classification layer logic.
Methodology
The authors propose two surgical modifications to the standard tagging architecture:
Focal Training
Instead of standard cross-entropy, they use Focal Loss. This loss function adds a modulating factor $(1 - p_t)^\gamma$ to the objective. When the model is confident about a "KEEP" tag (high $p_t$), the loss for that token is drastically down-weighted. This forces the gradient updates to focus on the "hard" tokens—the actual grammatical errors.
Tagger Decoupling
In standard GECToR, the error detection head and the label tagging head are just two linear layers sitting on top of the same encoder output. The authors introduce a Matching Block for the error detection head. This module uses a self-attention mechanism to create a "matched" representation ($M$) of the hidden states before passing them to the detection layer. This allows the detection task to have its own specialized view of the sentence context, separate from the tagging task.
Experimental Setup
The study focuses on the Chinese language, evaluating the model on three recent benchmarks:
- MuCGEC: A multi-source dataset covering diverse grammatical errors.
- FCGEC: A human-annotated corpus derived from Chinese school examinations (often more complex/formal).
- MCSCSet: A specialized dataset focused on Chinese Spelling Correction within the medical domain.
Results and Discussion
The proposed modifications consistently outperformed the GECToR baseline across all metrics. The results show that Focal Loss provides the initial boost by handling imbalance, while the Tagger Decoupling (TD) provides an additional, consistent gain by refining how the model perceives errors.
Key Insights
- The "Easy" Class Trap: The paper highlights that "KEEP" tags are not just frequent; they are "easy" for the model to learn. By using Focal Loss, the researchers essentially tell the model, "We know you're good at identifying correct words; now spend your energy on the tricky substitutions and deletions."
- Specialization Matters: Decoupling the heads suggests that "Is this wrong?" and "How do I fix this?" require different features. The self-attention matching module acts as a dedicated feature extractor for the detection task, preventing the tagging requirements from "polluting" the detection logic.
- The Gamma ($\gamma$) Trade-off: Increasing the $\gamma$ parameter in Focal Loss makes the model more aggressive toward errors. However, the authors found a "sweet spot" (usually $\gamma=2$). If $\gamma$ is too high, the model overfits the training distribution and loses its ability to generalize to different types of text.
Conclusion
This work demonstrates that addressing label imbalance and task entanglement can significantly enhance tagging-based GEC models. While the experiments were conducted exclusively on Chinese datasets, the logic should theoretically apply to other languages. Future work could involve testing these architectural changes on even larger pre-trained encoders and refining inference parameters to improve generalization.
Summary updated