Introduction
Standard sequence-to-sequence (seq2seq) models, such as the Transformer, are typically designed for tasks like machine translation where input and output sequences differ significantly. These models utilize a causal decoder that can only attend to previously generated tokens, effectively remaining "blind" to future context. However, in Local Sequence Transduction (LST) tasks—including Grammatical Error Correction (GEC), spell checking, and OCR correction—the input and output sequences are highly similar.
In LST, the source sentence often contains most of the information needed for the target. For instance, in correcting "He go to school," a standard decoder only sees "He" when predicting the next token. If it could access the future context "to school yesterday," it would more easily identify the correct tense ("went").
Methodology: Pseudo-Bidirectional Decoding (PBD)
To address the causal bottleneck without the computational overhead of true bidirectional decoding, the authors propose Pseudo-Bidirectional Decoding (PBD). This approach leverages the high similarity between source and target in LST tasks by using the source sentence as a "pseudo" version of the future target.
The core innovations include:
- Pseudo Future Modeling: The model copies representations of source tokens from the encoder and feeds them into the decoder's self-attention mechanism as future context.
- Novel Masking Strategy: A specific attention mask allows the model to attend to both previously generated "real" tokens and "pseudo" future tokens from the source sequence simultaneously during training.
- Parameter Sharing: Since PBD makes the decoder's role more similar to the encoder's (both process bidirectional context), the authors share all weights between the encoder and decoder. This reduces model size by approximately 50% and serves as a powerful regularizer.
- Segment Embeddings: Learned embeddings distinguish between "real" generated tokens and "pseudo" future tokens copied from the source.
Evaluation and Results
The model was evaluated on GEC (BEA-19, CoNLL-14), Twitter spell correction, and Finnish OCR datasets.
As shown in the results table, the PBD model (labeled "Ours") consistently outperforms standard Transformer baselines while using significantly fewer parameters. On the CoNLL-14 benchmark, the 132M parameter PBD model achieved an $F_{0.5}$ score of 53.7, surpassing the 210M parameter baseline (51.5). When scaled to 12 layers with pre-training, it reached state-of-the-art scores of 60.1 on CoNLL-14 and 63.9 on BEA-19.
Similar gains were observed in other domains. In OCR correction, the approach improved accuracy from 84.5% to 88.7% compared to a vanilla Transformer. Ablation studies confirmed that while parameter sharing is beneficial, the Pseudo Future Modeling provides the primary performance boost.
Discussion and Limitations
The core insight of this research is that in LST, the future context is already available in the input buffer. By treating the source as a proxy for the future, the model gains bidirectional benefits without increased latency.
However, the method is task-specific. It is optimized for "local" changes and may struggle with tasks like summarization or translation where source and target lengths and meanings diverge significantly. Furthermore, the current implementation relies on simple position-wise alignment; corrections involving significant reordering or many insertions/deletions might introduce noise into the "pseudo" context. Future work may explore more sophisticated alignment mechanisms to handle such cases.
Summary updated