Introduction
Standard sequence-to-sequence models, such as the Transformer, are inherently "causal" or unidirectional during decoding. They generate text strictly from left to right, meaning the decoder is "blind" to future context. While this is necessary for tasks like machine translation where the target structure differs significantly from the source, it represents a missed opportunity for Local Sequence Transduction (LST).
LST tasks—including Grammatical Error Correction (GEC), spell-checking, and OCR correction—are characterized by high overlap between input and output sequences. In these scenarios, the input sequence itself serves as a highly accurate proxy for the future context of the output. Ignoring this available information leads to suboptimal performance.
Methodology: Pseudo-Bidirectional Decoding (PBD)
This paper introduces Pseudo-Bidirectional Decoding (PBD), a framework designed to provide the decoder with a "look-ahead" at future tokens without the computational overhead of multi-pass decoding. The approach rests on four key innovations:
- Pseudo Future Modeling: The model copies input tokens from the current decoding position onward and feeds them into the decoder. These act as "pseudo-future" tokens, providing immediate context for the words yet to be generated.
- Pseudo-Bidirectional Attention Mask: A custom attention mask allows the model to attend simultaneously to previously generated "real" tokens and the "pseudo" tokens from the input. This maintains the parallel processing efficiency of the Transformer while enabling bidirectional context.
- Parameter Sharing: Given the similarity between the noisy input and clean output in LST, the authors share weights between the encoder and decoder. This acts as a strong regularizer and significantly reduces the model's footprint.
- Segment Embeddings: To help the model distinguish between generated tokens and input-derived pseudo-tokens, segment embeddings are utilized, similar to the architecture found in BERT.
Experimental Setup
The researchers evaluated PBD across three domains:
- Grammatical Error Correction (GEC): Benchmarked on BEA-19 and CoNLL-14 using standard datasets like Lang-8 and NUCLE.
- Spell Correction: Evaluated using Twitter-derived word pairs.
- OCR Correction: Tested on a Finnish OCR dataset of early modern text.
Key Results
The PBD approach consistently outperformed standard Transformer baselines while maintaining higher parameter efficiency.
As shown in the results, the "Ours-12 layers" model achieved a state-of-the-art $F_{0.5}$ score of 63.9 on BEA-19, outperforming a standard 12-layer Transformer (61.9) despite using significantly fewer parameters (232M vs 383M). Furthermore, the 132M parameter PBD model outperformed a much larger 210M parameter standard Transformer, demonstrating the power of weight sharing and pseudo-future context.
In specialized tasks, the gains were even more significant:
- OCR Correction: Accuracy improved from 85.4% to 89.6%.
- Spell Correction: Accuracy rose from 67.1% to 70.4%.
Conclusion
Pseudo-Bidirectional Decoding proves that for Local Sequence Transduction, the input is a computationally "free" and highly effective proxy for future context. By integrating this look-ahead capability and leveraging parameter sharing, PBD achieves superior accuracy with nearly half the parameters of standard models. While specifically tailored for LST, the method offers a blueprint for more efficient and context-aware sequence generation.
Summary updated