Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring

Hirofumi Inaguma, Yosuke Higuchi, Kevin Duh, Tatsuya Kawahara, Shinji Watanabe

Abstract

End-to-end speech translation (E2E-ST) models traditionally face a trade-off between the high accuracy of slow autoregressive (AR) models and the high speed of non-autoregressive (NAR) models. This paper introduces Orthros, a framework that bridges this gap by combining a fast NAR decoder with a shallow auxiliary AR decoder for parallel rescoring. Orthros achieves translation quality comparable to state-of-the-art AR models while providing significant speedups—up to 3.6x on standard CPUs.

Introduction

In AI-powered translation, speed and quality are often at odds. Traditional AR models generate text one token at a time, which is accurate but computationally expensive. Conversely, NAR models generate all tokens in parallel, which is fast but prone to the "multimodality problem": because tokens are predicted independently, the model may produce incoherent "Frankenstein" sentences by mixing parts of different valid translations.

To address this, researchers from Kyoto University, Johns Hopkins, and Carnegie Mellon proposed Orthros. This dual-decoder architecture maintains the speed of NAR models while employing a lightweight AR "safety net" to ensure linguistic coherence.

The Orthros Framework

Orthros (named after the two-headed dog of Greek mythology) utilizes a shared encoder and two distinct decoders to balance efficiency and accuracy.

Overview of the Orthros framework architecture, showing the shared speech encoder, NAR decoder, and the shallow AR decoder used for parallel rescoring.

Architecture Components

  1. Shared Encoder: A Conformer or Transformer backbone processes the input audio into high-level representations.
  2. NAR Decoder: The primary engine for rapid generation. The framework supports both Conditional Masked Language Models (CMLM), which use iterative refinement, and Connectionist Temporal Classification (CTC), which maps audio frames directly to output tokens in a single step.
  3. Auxiliary Shallow AR Decoder: A thin (often single-layer) AR decoder that performs Parallel Rescoring. During inference, the NAR decoder generates multiple candidate translations. Because these candidates are already fully formed, the AR decoder can score them simultaneously using the Transformer's self-attention mechanism, avoiding the latency of sequential generation.

The training process is further enhanced by Multi-Mask Training (MMT) and an auxiliary NAR machine translation task, which helps the NAR decoder better capture language patterns from text-only data.

Experimental Results

The models were evaluated on major benchmarks including MuST-C (English to five European languages), Libri-trans (English-French), and Fisher-CallHome (Spanish-English).

Performance comparison on the MuST-C benchmark using Conformer-based models. Orthros-CTC achieves competitive BLEU scores with significant CPU speedups.

Key Insights

  • Shallow is Sufficient: A surprising finding was that a very shallow AR decoder (even a single layer) is sufficient to significantly improve NAR output quality.
  • The Conformer Advantage: Utilizing a Conformer encoder—which uses convolutions to capture local audio patterns—dramatically improved CTC-based models, making them competitive with AR baselines for the first time.
  • Efficiency and Speed: Orthros-CTC achieved a 3.63x speedup on CPU compared to a standard AR Transformer. The parallel rescoring process adds minimal latency (5–100ms), making it highly practical for real-time deployment.
  • Robustness: The CTC-based variant demonstrated superior robustness when translating long-form speech (up to 60 seconds), where standard AR models often suffer from hallucinations or failure.

Conclusion

Orthros successfully bridges the performance gap between NAR and AR models for speech translation. While a small quality margin remains compared to the most complex AR models using large beam searches, the massive gains in inference speed and robustness to long inputs make Orthros a compelling framework for real-world applications. Future research will focus on deeper encoder architectures and refined candidate selection to further optimize the speed-quality trade-off.

Summary updated

Abstract

This article describes an efficient end-to-end speech translation (E2E-ST) framework based on non-autoregressive (NAR) models. End-to-end speech translation models have several advantages over traditional cascade systems such as inference latency reduction. However, conventional AR decoding methods are not fast enough because each token is generated incrementally. NAR models, however, can accelerate the decoding speed by generating multiple tokens in parallel on the basis of the token-wise conditional independence assumption. We propose a unified NAR E2E-ST framework called Orthros, which has an NAR decoder and an auxiliary shallow AR decoder on top of the shared encoder. The auxiliary shallow AR decoder selects the best hypothesis by rescoring multiple candidates generated from the NAR decoder in parallel (parallel AR rescoring). We adopt conditional masked language model (CMLM) and a connectionist temporal classification (CTC)-based model as NAR decoders for Orthros, referred to as Orthros-CMLM and Orthros-CTC, respectively. We also propose two training methods to enhance the CMLM decoder. Experimental evaluations on three benchmark datasets with six language directions demonstrated that Orthros achieved large improvements in translation quality with a very small overhead compared with the baseline NAR model. Moreover, the Conformer encoder architecture enabled large quality improvements, especially for CTC-based models. Orthros-CTC with the Conformer encoder increased decoding speed by 3.63x on CPU with translation quality comparable to that of an AR model.
BibTeX
@misc{inaguma2021non,
  author = {Hirofumi Inaguma and Yosuke Higuchi and Kevin Duh and Tatsuya Kawahara and Shinji Watanabe},
  title = {{Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoring}},
  year = {2021},
  eprint = {2109.04411},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.04411},
}
Figure