Pseudo-Bidirectional Decoding for Local Sequence Transduction

Wangchunshu Zhou, Tao Ge, Ke Xu

Introduction

Standard sequence-to-sequence models, such as the Transformer, are inherently "causal" or unidirectional during decoding. They generate text strictly from left to right, meaning the decoder is "blind" to future context. While this is necessary for tasks like machine translation where the target structure differs significantly from the source, it represents a missed opportunity for Local Sequence Transduction (LST).

LST tasks—including Grammatical Error Correction (GEC), spell-checking, and OCR correction—are characterized by high overlap between input and output sequences. In these scenarios, the input sequence itself serves as a highly accurate proxy for the future context of the output. Ignoring this available information leads to suboptimal performance.

Figure 1: Comparison between General Sequence Transduction (e.g., Translation) and Local Sequence Transduction (e.g., GEC and Spell Correction).

Methodology: Pseudo-Bidirectional Decoding (PBD)

This paper introduces Pseudo-Bidirectional Decoding (PBD), a framework designed to provide the decoder with a "look-ahead" at future tokens without the computational overhead of multi-pass decoding. The approach rests on four key innovations:

  1. Pseudo Future Modeling: The model copies input tokens from the current decoding position onward and feeds them into the decoder. These act as "pseudo-future" tokens, providing immediate context for the words yet to be generated.
  2. Pseudo-Bidirectional Attention Mask: A custom attention mask allows the model to attend simultaneously to previously generated "real" tokens and the "pseudo" tokens from the input. This maintains the parallel processing efficiency of the Transformer while enabling bidirectional context.
  3. Parameter Sharing: Given the similarity between the noisy input and clean output in LST, the authors share weights between the encoder and decoder. This acts as a strong regularizer and significantly reduces the model's footprint.
  4. Segment Embeddings: To help the model distinguish between generated tokens and input-derived pseudo-tokens, segment embeddings are utilized, similar to the architecture found in BERT.

Experimental Setup

The researchers evaluated PBD across three domains:

  • Grammatical Error Correction (GEC): Benchmarked on BEA-19 and CoNLL-14 using standard datasets like Lang-8 and NUCLE.
  • Spell Correction: Evaluated using Twitter-derived word pairs.
  • OCR Correction: Tested on a Finnish OCR dataset of early modern text.

Key Results

The PBD approach consistently outperformed standard Transformer baselines while maintaining higher parameter efficiency.

Figure 6: Performance comparison on GEC benchmarks (BEA-19 and CoNLL-14) showing parameter counts and the impact of pretraining.

As shown in the results, the "Ours-12 layers" model achieved a state-of-the-art $F_{0.5}$ score of 63.9 on BEA-19, outperforming a standard 12-layer Transformer (61.9) despite using significantly fewer parameters (232M vs 383M). Furthermore, the 132M parameter PBD model outperformed a much larger 210M parameter standard Transformer, demonstrating the power of weight sharing and pseudo-future context.

In specialized tasks, the gains were even more significant:

  • OCR Correction: Accuracy improved from 85.4% to 89.6%.
  • Spell Correction: Accuracy rose from 67.1% to 70.4%.

Conclusion

Pseudo-Bidirectional Decoding proves that for Local Sequence Transduction, the input is a computationally "free" and highly effective proxy for future context. By integrating this look-ahead capability and leveraging parameter sharing, PBD achieves superior accuracy with nearly half the parameters of standard models. While specifically tailored for LST, the method offers a blueprint for more efficient and context-aware sequence generation.

Summary updated

Abstract

Local sequence transduction (LST) tasks are sequence transduction tasks where there exists massive overlapping between the source and target sequences, such as Grammatical Error Correction (GEC) and spell or OCR correction. Previous work generally tackles LST tasks with standard sequence-to-sequence (seq2seq) models that generate output tokens from left to right and suffer from the issue of unbalanced outputs. Motivated by the characteristic of LST tasks, in this paper, we propose a simple but versatile approach named Pseudo-Bidirectional Decoding (PBD) for LST tasks. PBD copies the corresponding representation of source tokens to the decoder as pseudo future context to enable the decoder to attends to its bi-directional context. In addition, the bidirectional decoding scheme and the characteristic of LST tasks motivate us to share the encoder and the decoder of seq2seq models. The proposed PBD approach provides right side context information for the decoder and models the inductive bias of LST tasks, reducing the number of parameters by half and providing good regularization effects. Experimental results on several benchmark datasets show that our approach consistently improves the performance of standard seq2seq models on LST tasks.
BibTeX
@inproceedings{zhou2020pseudo,
  author = {Wangchunshu Zhou and Tao Ge and Ke Xu},
  title = {{Pseudo-Bidirectional Decoding for Local Sequence Transduction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.136},
}
Figure