Abstract
Attention-based neural speech recognition models are frequently decoded with beam search, which produces a tree of hypotheses. In many cases, such as when using external language models, numerous decoding hypotheses need to be considered, requiring large beam sizes during decoding. We demonstrate that it is possible to merge certain nodes in a tree of hypotheses, in order to obtain a decoding lattice, which increases the number of decoding hypotheses without increasing the number of candidates that are scored by the neural network. We propose a convolutional architecture, which facilitates comparing states of the model at different time steps. The experiments are carried on the Wall Street Journal dataset, where the lattice decoder obtains lower word error rates with smaller beam sizes, than an otherwise similar architecture with regular beam search.
Introduction
Most modern Automatic Speech Recognition (ASR) systems use beam search to find the most likely transcript. Beam search produces a "tree" of hypotheses: as the model predicts each new character, it keeps the top $N$ best candidates. However, this approach suffers from significant redundancy. Many paths in the tree might eventually lead to the same state, but because standard Recurrent Neural Network (RNN) decoders have "infinite" memory (their hidden state depends on every previous character), two paths are almost never exactly identical. This prevents the system from merging them, leading to a restricted search space unless computationally expensive large beam sizes are used.
Methods
The core innovation of this work is the transition from an RNN-based decoder to a Temporal Convolutional Network (TCN) decoder. Unlike RNNs, a TCN only looks at a fixed-size window of previous characters. This finite receptive field means that if two different hypotheses end with the same suffix, their decoder states become exactly identical.
The authors propose an algorithm that merges these identical states into a lattice (a Directed Acyclic Graph). Two nodes are merged if their TCN states match, their external language model states are the same, and their attention weights are sufficiently similar. This allows the system to compactly represent an exponential number of possible sentences.
Key Results
The models were evaluated on the Wall Street Journal (WSJ) dataset. The lattice decoder consistently outperformed standard beam search across various metrics:
- 3x Efficiency Boost: The lattice decoder achieved similar performance to a standard beam search while using a 3x smaller beam size (e.g., a lattice with beam 50 performed as well as a tree with beam 150).
- Lower Error Rates: For beam sizes between 20 and 150, the lattice approach consistently yielded lower Word Error Rates (WER). On the
Eval92set, the lattice decoder reached a WER of 6.98% compared to 7.16% for the standard beam. - Massive Hypothesis Space: With a beam size of 10, the proposed lattice contained an average of over 12,700 valid paths, providing a much richer search space for the same computational cost as a 10-hypothesis tree.
Conclusion
By leveraging the finite memory of TCNs, this research demonstrates that attention-based models can generate efficient decoding lattices. This technique significantly reduces the redundancy of beam search, allowing for much larger effective search spaces and lower error rates without increasing the number of neural network forward passes. This approach is particularly valuable for complex decoding tasks involving external language models or multi-pass processing.
Summary updated