Agreement on Target-Bidirectional LSTMs for Sequence-to-Sequence Learning

Lemao Liu, Andrew M. Finch, Masao Utiyama, Eiichiro SumitaAAAI 2016 · DOI

Abstract: Standard sequence-to-sequence (seq2seq) models using unidirectional Long Short-Term Memory (LSTM) networks often suffer from performance degradation as sequence length increases, particularly at the end of the generated sequence. This paper proposes a target-bidirectional agreement mechanism that jointly trains and decodes two LSTMs—one generating left-to-right and the other right-to-left. By enforcing agreement between these models, the system leverages the prefix-strength of the L2R model and the suffix-strength of the R2L model. Evaluated on transliteration and grapheme-to-phoneme tasks, the proposed Bidirectional Ensemble LSTM (BELSTM) significantly outperforms unidirectional baselines and previous state-of-the-art hybrid systems.

1. Introduction

Recurrent Neural Networks (RNNs) have become the standard for sequence-to-sequence learning. However, a fundamental shortcoming exists in unidirectional LSTMs: as they generate sequences, they rely on their own previous predictions as context. Early errors propagate, "poisoning" the context for subsequent steps and leading to "unbalanced targets" where the prefix is accurate but the suffix is poor. Empirically, for sequences longer than 10 characters, precision drops from approximately 77% at the start to 65% at the end. This paper addresses this by introducing a bidirectional agreement mechanism for target generation.

2. The Agreement Model

The core innovation is the use of a pair of LSTMs: a Left-to-Right (L2R) model and a Right-to-Left (R2L) model. These models are jointly trained to maximize the product of their probabilities, ensuring that both the beginning and the end of a sequence are high-quality.

The bidirectional agreement mechanism for target sequence generation.

2.1 Joint Training and Decoding

Finding the optimal sequence satisfying both models is NP-hard. The authors propose two approximate search methods:

  • Joint $k$-best Approximation: Combines top-$k$ candidates from both models and re-scores them using the joint agreement score.
  • Polynomial Approximation: A more complex method that constructs a search space by concatenating prefixes from L2R candidates with suffixes from R2L candidates.

3. Experimental Setup

The approach was evaluated on two transduction tasks:

  • Machine Transliteration: Japanese (Katakana) to English conversion using ~59,000 pairs from Wikipedia.
  • Grapheme-to-Phoneme (G2P): Spelling to pronunciation conversion using the CMUdict dataset (~110,000 pairs).

4. Results and Discussion

The bidirectional ensemble model (BELSTM) consistently outperformed all baselines, including complex hybrid architectures like neural nets combined with Weighted Finite State Transducers.

Comparative performance of the proposed BELSTM against baseline and state-of-the-art systems.

Key findings include:

  • Directional Complementarity: The L2R model excels at prefixes while the R2L model excels at suffixes. Forcing agreement leverages the high-confidence regions of both.
  • Search Efficiency: The "Joint $k$-best" search was found to be nearly optimal, with 99.8% of errors attributed to model limitations rather than search algorithms.
  • Long Sequence Performance: Gains were most significant for targets longer than 10 tokens, with an absolute accuracy improvement of nearly 8% over unidirectional baselines.

5. Conclusion

By introducing target-bidirectional agreement, this work provides a robust solution to the "unbalanced target" problem in seq2seq learning. While it introduces some computational overhead, the gains in accuracy and the simplicity of the end-to-end neural approach make it a compelling alternative to hybrid systems. Future work may explore parameter sharing and scaling to full-sentence machine translation.

Summary updated

BibTeX
@inproceedings{liu2016agreement,
  author = {Lemao Liu and Andrew M. Finch and Masao Utiyama and Eiichiro Sumita},
  title = {{Agreement on Target-Bidirectional LSTMs for Sequence-to-Sequence Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2016},
  doi = {10.1609/aaai.v30i1.10327},
}
Figure