S2ST-Omni: An Efficient and Scalable Multilingual Speech-to-Speech Translation Framework via Seamless Speech-Text Alignment and Streaming Speech Generation

Yu Pan, Yuguang Yang, Yanni Hu, Jianhao Ye, Xiang Zhang, Hongbin Zhou, Lei Ma, Jianjun Zhao

Abstract

Despite recent advances in multilingual speech-to-speech translation (S2ST), several critical challenges persist: 1) achieving high-quality translation remains a major hurdle, and 2) most existing methods heavily rely on large-scale parallel speech corpora, which are costly and difficult to obtain. To address these issues, we propose S2ST-Omni, an efficient and scalable framework for multilingual S2ST. Specifically, we decompose the S2ST task into speech-to-text translation (S2TT) and text-to-speech synthesis (TTS). For S2TT, we propose an effective speech language model that integrates the pretrained Whisper encoder for robust audio understanding and Qwen 3.0 for advanced text comprehension. A lightweight speech adapter is employed to bridge the modality gap between speech and text representations. To further facilitate multimodal knowledge learning, a two-stage fine-tuning strategy is introduced. In the TTS stage, we adopt a streaming autoregressive generation approach to produce natural and fluent target speech. Experiments on the CVSS benchmark show that S2ST-Omni consistently outperforms existing state-of-the-art S2ST systems in translation quality, highlighting its effectiveness and superiority.

Introduction

In the rapidly evolving landscape of AI-driven communication, Speech-to-Speech Translation (S2ST) remains a significant challenge. It requires a model to not only understand spoken nuances across diverse languages but also to translate that meaning accurately and synthesize it into natural-sounding speech. Current S2ST systems generally fall into two categories: End-to-End (E2E) models, which map source speech directly to target speech but are often data-hungry and difficult to debug, and Compositional models, which split the task into Speech-to-Text Translation (S2TT) and Text-to-Speech (TTS).

S2ST-Omni addresses the limitations of existing compositional models—which often fail to capture deep semantic relationships—by rethinking how Large Language Models (LLMs) are adapted for speech tasks. It introduces a framework that bridges a Whisper speech encoder with a Qwen 3.0 LLM using a novel "local-then-global" hybrid adapter and hierarchical language-aware features.

Methodology

The S2ST-Omni framework is built upon three innovative pillars designed to improve representation and translation accuracy:

  1. The Hybrid Speech Adapter: Instead of a simple linear projection, S2ST-Omni uses a "local-then-global" hierarchy. It first employs depthwise separable convolutions to capture local phonetic patterns, then downsamples the data to optimize compute, and finally applies Transformer blocks to model long-range semantic dependencies. This creates a cleaner abstraction for the LLM to process.
  2. Hierarchical Language-Aware (HLA) Augmentation: To address "language-blindness," the model injects source-language information at two levels:
  3. Acoustic Level (LA-Dual-CTC): Connectionist Temporal Classification (CTC) heads are conditioned on the source language using FiLM-style feature modulation, guiding acoustic layers with language-specific biases.
  4. Linguistic Level (LAP): The system dynamically generates prompts for the LLM (e.g., "The following is French speech. Translate it accurately into English.") to activate its internal multilingual knowledge.
  5. Progressive Fine-Tuning (PFT): To ensure stability when training a massive LLM with a new adapter, a two-stage approach is used. Stage I freezes the LLM and Encoder to train only the adapter for modality alignment. Stage II introduces LoRA (Low-Rank Adaptation) to the LLM to refine translation quality.
Overview of the S2ST-Omni architecture and its two-stage progressive fine-tuning strategy.

Experimental Results

The model was evaluated on the CVSS-C benchmark, focusing on three representative directions: French, Spanish, and German to English. S2ST-Omni set new benchmarks, significantly outperforming existing state-of-the-art systems.

Performance comparison of S2ST-Omni against state-of-the-art baselines on the CVSS-C benchmark.

As shown in the results, S2ST-Omni outperformed the strongest baseline (StreamSpeech) by an average of +24% BLEU and +31% ASR-BLEU. Notably, in the challenging German-to-English (De→En) direction, S2ST-Omni achieved 33.34 BLEU, a massive relative gain over previous methods. Ablation studies confirmed that both the acoustic-level (CTC) and linguistic-level (Prompting) language awareness are complementary and essential for peak performance.

Conclusion

S2ST-Omni represents a significant shift toward "SpeechLLMs" that truly understand the structural nuances of the languages they translate. By combining robust audio understanding with advanced text comprehension and a modular TTS backend, the framework provides a scalable solution for high-quality multilingual speech translation.

Summary updated

Abstract

Despite recent advances in multilingual speech-to-speech translation (S2ST), several critical challenges persist: 1) achieving high-quality translation remains a major hurdle, and 2) most existing methods heavily rely on large-scale parallel speech corpora, which are costly and difficult to obtain. To address these issues, we propose \textit{S2ST-Omni}, an efficient and scalable framework for multilingual S2ST. Specifically, we decompose the S2ST task into speech-to-text translation (S2TT) and text-to-speech synthesis (TTS). For S2TT, we propose an effective speech language model that integrates the pretrained Whisper encoder for robust audio understanding and Qwen 3.0 for advanced text comprehension. A lightweight speech adapter is employed to bridge the modality gap between speech and text representations. To further facilitate the multimodal knowledge learning, a two-stage fine-tuning strategy is introduced. In the TTS stage, we adopt a streaming autoregressive generation approach to produce natural and fluent target speech. Experiments on the CVSS benchmark show that S2ST-Omni consistently outperforms existing state-of-the-art S2ST systems in translation quality, highlighting its effectiveness and superiority.
BibTeX
@inproceedings{pan2026s,
  author = {Yu Pan and Yuguang Yang and Yanni Hu and Jianhao Ye and Xiang Zhang and Hongbin Zhou and Lei Ma and Jianjun Zhao},
  title = {{S2ST-Omni: An Efficient and Scalable Multilingual Speech-to-Speech Translation Framework via Seamless Speech-Text Alignment and Streaming Speech Generation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1004},
}
Figure