A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies

Florian Boyer, Yusuke Shinohara, Takaaki Ishii, Hirofumi Inaguma, Shinji Watanabe

TL;DR

This paper introduces a major extension to the ESPnet toolkit for Recurrent Neural Network Transducer (RNN-T) models, featuring flexible "free-form" architecture definitions, multi-task learning with auxiliary losses, and advanced decoding strategies. Notably, the authors propose a novel N-step Constrained Beam Search (NSC) that significantly improves decoding speed and accuracy, achieving state-of-the-art results on benchmark datasets like AISHELL-1 and LibriSpeech.

What Problem Does This Paper Address?

End-to-end Automatic Speech Recognition (ASR) has seen massive shifts toward transducer-based models (like RNN-T) due to their competitive performance and natural suitability for streaming applications. However, standard transducers suffer from two key challenges:

  1. Unrestricted Label Expansion: During both training and decoding, a transducer can emit an arbitrary number of text labels at a single acoustic timestep. This behavior breaks the ability to vectorize computations efficiently, severely bottlenecking decoding speed.
  2. Encoder Underfitting: Because the decoder network acts as a powerful language model, the encoder network can easily underfit during training, limiting the model's overall generalization capability.

While several open-source toolkits support transducers, none comprehensively address these issues with joint auxiliary training and optimized, flexible beam search strategies.

Methods

The authors extend the ESPnet toolkit to provide a modular framework for transducer models, introducing two main features:

1. Augmented Training via Multi-Task Learning

To mitigate encoder underfitting and regularize the search space, the architecture is augmented with four auxiliary classifier layers. The total loss function ($\mathcal{L}_{\text{tot}}$) jointly optimizes five distinct objectives:

  • Main Transducer Loss ($\mathcal{L}_{\text{trans}}$)
  • CTC Loss ($\mathcal{L}_{\text{CTC}}$): Acts as a soft regularization to enforce monotonicity.
  • Auxiliary Transducer Loss ($\mathcal{L}_{\text{aux-trans}}$): Injected at intermediate encoder layers to boost gradient signals.
  • Symmetric KL-Divergence ($\mathcal{L}_{\text{symm-KL}}$): Penalizes inconsistent gradients between the main and auxiliary tasks.
  • Language Model Loss ($\mathcal{L}_{\text{LM}}$): Regularizes the decoder using cross-entropy with label smoothing.
Fig. 1. Transducer architecture with auxiliary tasks. In black, the original structure. In red, the new losses, connexions and intermedi- ate networks.

2. Advanced Decoding Strategies

The paper evaluates four beam search strategies, including a newly proposed N-step Constrained Beam Search (NSC). NSC builds upon One-Step Constrained (OSC) search but expands the constraint to $N$ steps through an additional loop. Crucially, it introduces a counting method (auto-Nstep) that allows incomplete hypotheses to pass to the next timestep without prematurely penalizing them with a blank transition score, avoiding the deletion of viable search paths.

Datasets

The framework was trained and evaluated across three distinct datasets:

  • Voxforge Italian: A relatively small corpus (~20 hours) used primarily for ablation studies and decoding strategy benchmarks.
  • AISHELL-1: A well-known Mandarin speech corpus used to evaluate Character Error Rate (CER).
  • LibriSpeech: A standard English audiobooks corpus (960 hours) used to evaluate Word Error Rate (WER).

Key Insights and Ideas

  • CTC as a Soft Regularizer: Integrating an auxiliary CTC loss forces the model to behave more monotonically. This significantly reduces the number of multi-label expansions per timestep during decoding (shifting the distribution heavily toward single-label emissions), which directly translates to faster, more predictable inference.
  • The Danger of Over-Constraining: While restricting emissions to a single token per frame (like original OSC decoding) speeds up inference, it degrades accuracy on low-resource tasks where the model may genuinely need multiple expansions per frame to resolve acoustics. NSC successfully balances this trade-off.
  • Initialization vs. Auxiliary Tasks: Pre-initializing an encoder with a pre-trained CTC model helps, but training jointly with an auxiliary CTC loss yields a better balance between character accuracy (CER) and word accuracy (WER).

Key Results

The additions allow ESPnet's transducer implementations to match or exceed state-of-the-art attention-based models.

  • AISHELL-1: The Conformer-Transducer augmented with auxiliary tasks (Conformer-T + Aux) achieved a 4.3% Dev / 4.7% Test CER, outperforming previous transducer results in literature and matching the top attention-based Conformer models.
  • LibriSpeech: Switching from a standard RNN-T to a Conformer-T combined with auxiliary losses and language model shallow fusion yielded a highly competitive 2.6% (test-clean) / 6.1% (test-other) WER.
  • Decoding Efficiency: As shown in the CER vs. RTF (Real-Time Factor) trade-off analysis, the proposed NSC decoding strategy provides a significantly better accuracy-to-speed profile than standard Time-Synchronous Decoding (TSD), especially when models are trained with auxiliary losses.
CER against RTF for the decoding strategies.

Limitations and Future Work

While this work significantly advances the usability of transducers in ESPnet, the authors note a few areas currently in active development:

  • Streaming Support: Although the decoding strategies lay the groundwork for streaming, full online streaming recipes and benchmarks are left for future releases.
  • Discriminative Training: Incorporating Minimum Bayes Risk (MBR) training directly into the transducer pipeline is an ongoing effort.
  • Comparison to Non-Autoregressive Models: Future work will include a deeper analysis contrasting these optimized transducers against emerging non-autoregressive ASR architectures.

Summary updated

Abstract

In this study, we present recent developments of models trained with the RNN-T loss in ESPnet. It involves the use of various architectures such as recently proposed Conformer, multi-task learning with different auxiliary criteria and multiple decoding strategies, including our own proposition. Through experiments and benchmarks, we show that our proposed systems can be competitive against other state-of-art systems on well-known datasets such as LibriSpeech and AISHELL-1. Additionally, we demonstrate that these models are promising against other already implemented systems in ESPnet in regards to both performance and decoding speed, enabling the possibility to have powerful systems for a streaming task. With these additions, we hope to expand the usefulness of the ESPnet toolkit for the research community and also give tools for the ASR industry to deploy our systems in realistic and production environments.
BibTeX
@inproceedings{boyer2021study,
  author = {Florian Boyer and Yusuke Shinohara and Takaaki Ishii and Hirofumi Inaguma and Shinji Watanabe},
  title = {{A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies}},
  booktitle = {ASRU},
  year = {2021},
  doi = {10.1109/asru51503.2021.9688251},
}
Figure