Abstract
Sequence-to-sequence models with soft attention have achieved significant success in tasks like machine translation and speech recognition. However, these models typically require the entire input sequence to be available before inference begins, making them unsuitable for real-time applications. We present a new method for online sequence-to-sequence learning using hard binary stochastic decisions to select output timesteps. Trained via policy gradients with variance reduction and entropy regularization, our model achieves competitive performance on the TIMIT and Wall Street Journal (WSJ) speech recognition datasets, demonstrating the feasibility of high-accuracy, low-latency online alignments.
Introduction: The Latency Bottleneck
Standard sequence-to-sequence (seq2seq) architectures rely on global soft attention. While effective, soft attention is an "offline" process: the model must process the complete input sequence to compute attention weights before generating the first output token. This creates a prohibitive latency bottleneck for real-time applications like instantaneous voice translation.
To address this, we propose an "online" model that makes "hard" decisions: at each input timestep, the model must decide whether to emit an output token or wait for more context. This requires the model to learn temporal alignments without seeing the future.
Methodology
The core of our approach is a recurrent neural network (RNN) that produces a binary stochastic decision at every input timestep.
The Decision Mechanism
At each step $i$, the model outputs a probability $b_i$, from which a binary variable $\tilde{b}_i \in \{0, 1\}$ is sampled. If $\tilde{b}_i = 1$, the model generates an output token; if $\tilde{b}_i = 0$, it consumes the next input frame.
Policy Gradient Training
Since the sampling process is non-differentiable, we employ the REINFORCE policy gradient algorithm. The reward signal is defined as the log-probability of the target sequence. To stabilize training and reduce variance, we utilize:
- Baselines: Centering rewards to reduce gradient swings.
- Rao-Blackwellization: Restricting gradient calculations to future rewards to improve estimator efficiency.
The Critical Role of Regularization
A key challenge in training hard alignments is the tendency for the model to converge to degenerate solutions, such as emitting all tokens at the very beginning or end of a sequence. We found that entropy regularization is essential for success. By penalizing low-entropy distributions, we force the model to explore different emission timings, eventually learning to spread outputs across the timeline in a way that reflects the underlying audio-text relationship.
Experimental Results
We evaluated the model on two speech recognition benchmarks:
- TIMIT: Using a Grid LSTM, the model achieved a Phoneme Error Rate (PER) of 20.5%. This is highly competitive with offline models like CTC (19.6%), proving that the online constraint does not necessarily sacrifice significant accuracy.
- Wall Street Journal (WSJ): On this larger 81-hour dataset, the model reached a PER of 27.0%. While trailing state-of-the-art bidirectional offline models (~18.6%), it serves as a proof-of-concept for scaling policy gradient-based online alignments to large-vocabulary tasks.
Qualitative analysis of the learned alignments shows that the model effectively "clears" its internal memory once sufficient information is gathered to make a confident prediction, emitting tokens only when relevant new audio features arrive.
Conclusion and Future Work
This work demonstrates that a single binary decision per timestep is sufficient for training accurate online seq2seq models. While a performance gap remains between online and offline models—largely due to the inability of online models to use bidirectional context—future improvements in policy gradient methods and model scaling offer a path toward closing this gap for real-time translation and recognition systems.
Summary updated