Learning Online Alignments with Continuous Rewards Policy Gradient

Yuping Luo, Chung-Cheng Chiu, Navdeep Jaitly, Ilya Sutskever

Abstract

Sequence-to-sequence models with soft attention have achieved significant success in tasks like machine translation and speech recognition. However, these models typically require the entire input sequence to be available before inference begins, making them unsuitable for real-time applications. We present a new method for online sequence-to-sequence learning using hard binary stochastic decisions to select output timesteps. Trained via policy gradients with variance reduction and entropy regularization, our model achieves competitive performance on the TIMIT and Wall Street Journal (WSJ) speech recognition datasets, demonstrating the feasibility of high-accuracy, low-latency online alignments.

Introduction: The Latency Bottleneck

Standard sequence-to-sequence (seq2seq) architectures rely on global soft attention. While effective, soft attention is an "offline" process: the model must process the complete input sequence to compute attention weights before generating the first output token. This creates a prohibitive latency bottleneck for real-time applications like instantaneous voice translation.

To address this, we propose an "online" model that makes "hard" decisions: at each input timestep, the model must decide whether to emit an output token or wait for more context. This requires the model to learn temporal alignments without seeing the future.

Methodology

The core of our approach is a recurrent neural network (RNN) that produces a binary stochastic decision at every input timestep.

The Decision Mechanism

At each step $i$, the model outputs a probability $b_i$, from which a binary variable $\tilde{b}_i \in \{0, 1\}$ is sampled. If $\tilde{b}_i = 1$, the model generates an output token; if $\tilde{b}_i = 0$, it consumes the next input frame.

Architecture of the online sequence-to-sequence model with binary stochastic decisions.

Policy Gradient Training

Since the sampling process is non-differentiable, we employ the REINFORCE policy gradient algorithm. The reward signal is defined as the log-probability of the target sequence. To stabilize training and reduce variance, we utilize:

  • Baselines: Centering rewards to reduce gradient swings.
  • Rao-Blackwellization: Restricting gradient calculations to future rewards to improve estimator efficiency.

The Critical Role of Regularization

A key challenge in training hard alignments is the tendency for the model to converge to degenerate solutions, such as emitting all tokens at the very beginning or end of a sequence. We found that entropy regularization is essential for success. By penalizing low-entropy distributions, we force the model to explore different emission timings, eventually learning to spread outputs across the timeline in a way that reflects the underlying audio-text relationship.

Comparison of alignment patterns under different regularization schemes. Without regularization, the model fails to learn meaningful temporal alignments.

Experimental Results

We evaluated the model on two speech recognition benchmarks:

  1. TIMIT: Using a Grid LSTM, the model achieved a Phoneme Error Rate (PER) of 20.5%. This is highly competitive with offline models like CTC (19.6%), proving that the online constraint does not necessarily sacrifice significant accuracy.
  2. Wall Street Journal (WSJ): On this larger 81-hour dataset, the model reached a PER of 27.0%. While trailing state-of-the-art bidirectional offline models (~18.6%), it serves as a proof-of-concept for scaling policy gradient-based online alignments to large-vocabulary tasks.

Qualitative analysis of the learned alignments shows that the model effectively "clears" its internal memory once sufficient information is gathered to make a confident prediction, emitting tokens only when relevant new audio features arrive.

Conclusion and Future Work

This work demonstrates that a single binary decision per timestep is sufficient for training accurate online seq2seq models. While a performance gap remains between online and offline models—largely due to the inability of online models to use bidirectional context—future improvements in policy gradient methods and model scaling offer a path toward closing this gap for real-time translation and recognition systems.

Summary updated

Abstract

Sequence-to-sequence models with soft attention had significant success in machine translation, speech recognition, and question answering. Though capable and easy to use, they require that the entirety of the input sequence is available at the beginning of inference, an assumption that is not valid for instantaneous translation and speech recognition. To address this problem, we present a new method for solving sequence-to-sequence problems using hard online alignments instead of soft offline alignments. The online alignments model is able to start producing outputs without the need to first process the entire input sequence. A highly accurate online sequence-to-sequence model is useful because it can be used to build an accurate voice-based instantaneous translator. Our model uses hard binary stochastic decisions to select the timesteps at which outputs will be produced. The model is trained to produce these stochastic decisions using a standard policy gradient method. In our experiments, we show that this model achieves encouraging performance on TIMIT and Wall Street Journal (WSJ) speech recognition datasets.
BibTeX
@inproceedings{luo2017learning,
  author = {Yuping Luo and Chung-Cheng Chiu and Navdeep Jaitly and Ilya Sutskever},
  title = {{Learning Online Alignments with Continuous Rewards Policy Gradient}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=zoThhB9o_Ur},
}
Figure