This paper presents a significant step toward unifying training and inference in sequence-to-sequence tasks like Automatic Speech Recognition (ASR). By introducing a Fully Differentiable Beam Search Decoder (DBD), the authors allow gradients to flow directly through the decoding process, enabling the joint optimization of acoustic and language models.
The Disconnect in Traditional ASR Pipelines
Traditional speech recognition pipelines often suffer from three main structural issues:
- Exposure Bias: Models are typically trained using ground-truth targets (a technique known as "teacher forcing"). However, during inference, they must rely on their own, potentially erroneous, predictions. This mismatch can lead to rapid error accumulation.
- Label Bias: Many models use local normalization, meaning they decide the next token based only on the current state. This prevents them from easily "revising" past decisions when new, contradictory future evidence appears.
- Disjoint Training: Acoustic models (AM) and language models (LM) are usually trained separately. Combining them requires a tedious "grid search" on a validation set to find the optimal weights for the LM and word insertion penalties.
The DBD aims to solve these issues by making the entire beam search—the process of finding the most likely word sequence—part of the differentiable training loop.
The Differentiable Beam Search Approach
The core innovation of this paper is replacing the non-differentiable max() operations typically found in Viterbi or beam search algorithms with the differentiable logadd() (or log-sum-exp) operation.
Sequence-Level Differentiable Loss
The model optimizes a sequence-level conditional likelihood. The loss function is essentially the difference between the total score of all valid alignments for the target transcription (the "numerator") and the total score of all hypotheses explored in the beam (the "denominator").
Joint Optimization and Implementation
Because the decoder is fully differentiable, the system can learn the optimal scaling for the acoustic scores, token transitions, and word-level language models simultaneously. To make this computationally feasible, the authors implemented a custom C++ autograd version of the decoder, which runs roughly 10x faster than standard deep learning framework implementations.
Key Insights and Advantages
- Automatic Hyperparameter Tuning: Because the LM weights are learned directly during training, DBD eliminates the tedious grid search phase usually required to tune a decoder.
- Efficiency through Constraints: By constraining the search to a valid lexicon during training, the acoustic model doesn't have to "waste" probability mass on gibberish sequences. This allows simpler acoustic models to achieve better results.
- Smaller Beams Work Better: Models trained with DBD are "aware" of the beam search process. Consequently, they perform better with much smaller beam sizes (e.g., 500) than traditional models, which might require beams of 8,000 or more to reach peak accuracy.
- Global Normalization: By using unnormalized scores and a global (sequence-level) normalization term, the model naturally avoids the label bias problem.
Key Results and Performance
The researchers evaluated their approach using the Wall Street Journal (WSJ) corpus, a standard benchmark in speech recognition containing approximately 81 hours of transcribed audio. The DBD approach showed clear advantages in both accuracy and model efficiency:
- Accuracy: On the WSJ
nov92test set, a 7.5M parameter acoustic model trained with DBD achieved a 5.1% Word Error Rate (WER). For comparison, a larger 10M parameter baseline using the Auto Segmentation Criterion (ASG) achieved only 5.6%, even with a much larger beam. - Beam Size Efficiency: DBD-trained models maintained high performance with a beam size of 500, whereas ASG models saw significant performance degradation when the beam was reduced from 8,000 to 500.
- Rapid Convergence: DBD models can be "bootstrapped" from pre-trained ASG models and quickly converge to better error rates.
Limitations and Future Work
While promising, the approach does have limitations. Training with DBD is significantly slower than simpler criteria; a single epoch on WSJ takes about 30 minutes with DBD compared to just a few minutes with ASG. Furthermore, the authors noted that when starting DBD training from a very late-stage pre-trained model, the system tended to overfit the training data. Finally, writing a differentiable decoder is non-trivial, requiring a custom scalar-based autograd engine to ensure gradient correctness.
Future work could involve scaling this to much larger datasets and exploring more complex, non-linear language models within the differentiable framework.
Summary updated