Abstract
Sequence-to-sequence (seq2seq) models have significantly advanced natural language processing but frequently produce outputs that are semantically plausible yet grammatically flawed. This paper presents an empirical study on utilizing a state-of-the-art Grammatical Error Correction (GEC) system as a modular post-processing step for various text generation tasks. Our findings indicate that while GEC significantly enhances grammaticality—particularly in low-resource or formal contexts—it introduces a "formality bias" that can be detrimental to tasks requiring informal outputs.
Introduction
Modern NLP relies heavily on seq2seq architectures for tasks such as machine translation and summarization. However, these models are essentially probabilistic engines that do not inherently encode grammatical rules, leading to the "fluent but wrong" phenomenon where outputs may miss prepositions or mangle verb tenses. Rather than modifying core model architectures, this study investigates whether a dedicated GEC system can serve as an effective, task-agnostic post-editor to refine seq2seq outputs.
Methodology
The researchers employed a 7-layer convolutional seq2seq GEC model, trained with a "fluency boost" strategy, which has demonstrated human-level performance on standard benchmarks. This model was integrated into a pipeline as a post-editor for the outputs of several base models, including Google Translate and standard Transformers.
The evaluation spanned four primary tasks:
- Machine Translation (MT): English paired with French, German, and Chinese (WMT14/17).
- Formality Style Transfer: Converting informal to formal text and vice versa using the GYAFC corpus.
- Sentence Compression: Shortening sentences while preserving meaning (GoogComp).
- Sentence Simplification: Reducing linguistic complexity (Newsela).
Results and Analysis
The Formality Bias
A critical finding of this study is the inherent formality bias of GEC systems. In style transfer tasks, GEC significantly improved performance when moving from Informal → Formal, with accuracy increasing from 83.0% to 84.2%. Conversely, for Formal → Informal transfer, style accuracy plummeted from 68.7% to 47.1%. Because GEC models are trained on "correct" (typically formal) English, they tend to strip away the informalisms intended by the base model.
Machine Translation and Low-Resource Scenarios
GEC proved most beneficial for unsupervised or low-resource MT models, which often struggle with basic fluency. For unsupervised MT, GEC improved BLEU scores by 0.22 to 0.24 points. Human evaluators corroborated these gains, noting improved grammaticality in 74% to 91% of cases across all tasks.
Metric Discrepancies
The study highlights that standard metrics like BLEU can be deceptive. GEC often improved a sentence's grammar while lowering its BLEU score. This occurs when GEC corrects a model's output to a form that is grammatically superior but differs from the specific wording in the human "gold standard" reference. Furthermore, the researchers found that human references in tasks like sentence compression often contain their own grammatical errors; by "fixing" these, GEC moves away from the reference, resulting in lower automated scores despite higher actual quality.
Discussion and Conclusion
While GEC is a powerful tool for enhancing the grammaticality of seq2seq models, it is not a universal solution. Its current "one-size-fits-all" approach to grammar correction makes it unsuitable for tasks requiring slang, dialect, or informal speech. Future research should focus on "task-aware GEC" systems that can adapt their correction strategies based on the intended style or domain of the generation task. Additionally, integrating GEC directly into the decoding process, rather than as a separate post-editing step, remains a promising avenue for improving the robustness of text generation models.
Summary updated