Abstract
We present a Parallel Iterative Edit (PIE) model for the problem of local sequence transduction arising in tasks like Grammatical error correction (GEC). Recent approaches are based on the popular encoder-decoder (ED) model for sequence to sequence learning. The ED model auto-regressively captures full dependency among output tokens but is slow due to sequential decoding. The PIE model does parallel decoding, giving up the advantage of modelling full dependency in the output, yet it achieves accuracy competitive with the ED model for four reasons:
- predicting edits instead of tokens, 2. labeling sequences instead of generating sequences, 3. iteratively refining predictions to capture dependencies, and 4. factorizing logits over edits and their token argument to harness pre-trained language models like BERT. Experiments on tasks spanning GEC, OCR correction and spell correction demonstrate that the PIE model is an accurate and significantly faster alternative for local sequence transduction.
Introduction
Most sequence-to-sequence (seq2seq) tasks, such as machine translation, are handled by auto-regressive Encoder-Decoder (ED) models. These models generate text one token at a time, where each new word depends on every word that preceded it. While powerful, this sequential nature is inherently slow, particularly for long sentences.
However, many transduction tasks are "local"—the input and output are largely identical, differing only in a few specific positions. Grammatical Error Correction (GEC), spell checking, and OCR correction fall into this category. For these tasks, using a massive, slow translation model is often overkill. The Parallel Iterative Edit (PIE) model reimagines these tasks as a parallel labeling problem rather than a sequential generation one, achieving accuracy competitive with state-of-the-art models while running 5 to 15 times faster.
The PIE Model Methodology
The PIE model rests on four design pillars that allow it to bypass the bottlenecks of traditional sequential decoding:
- Predicting Edits, Not Tokens: Instead of selecting from a vocabulary of 30,000+ words, the model predicts a small set of edit operations:
Copy,Delete,Append,Replace, andTransform(e.g., changing "run" to "running"). This simplifies the output space and generalizes better across different contexts. - Sequence Labeling: PIE treats the task as a labeling problem, assigning an edit tag to every token in the input sentence. To handle insertions without changing sequence length during the labeling phase, the model uses "compound edits" that merge an insertion with the preceding token.
- Iterative Refinement: To capture dependencies between words (e.g., ensuring subject-verb agreement), the model doesn't just run once. It feeds its own output back in as input for several rounds (typically 2-4), allowing it to "polish" the sentence iteratively.
- Factorized BERT Logits: The model leverages pre-trained bidirectional transformers (BERT). A custom architecture allows BERT to "warm-start" the training of edits by factorizing the logic, helping the model understand the relationship between the original token and the proposed edit.
Experimental Results
The model was evaluated across three distinct local transduction tasks: Grammatical Error Correction (using Lang-8, NUCLE, and FCE corpora), Spell Correction (Twitter dataset), and OCR Error Correction (Early Modern Finnish newspapers).
Accuracy and Efficiency
On the CoNLL-2014 benchmark, PIE achieved an $F_{0.5}$ score of 59.7, nearly identical to the state-of-the-art sequential models of the time (61.2) but at a fraction of the computational cost. The iterative process proved highly efficient; in testing, the vast majority of corrections occurred in the first round, with only a negligible number of sentences requiring a fourth round of editing.
Speed Scaling
Traditional auto-regressive models have a decoding time that increases linearly with the length of the sentence. In contrast, PIE’s decoding time is nearly constant (sub-linear), making it exponentially more efficient for long-form content. Overall, PIE is 5x to 15x faster than competitive Encoder-Decoder models like the Transformer-based T2T.
Conclusion
PIE demonstrates that for local sequence transduction, parallel labeling is a viable and highly efficient alternative to sequential generation. By recognizing that GEC is a labeling task, the model can look at both "future" and "past" contexts simultaneously. While PIE is specifically designed for local tasks and might struggle with global transformations like summarization, it sets a new standard for speed and efficiency in grammar and spell correction. Future work may involve applying this framework to more complex transduction tasks and further optimizing the iterative refinement process.
Summary updated