Abstract
Despite recent advances in multilingual speech-to-speech translation (S2ST), several critical challenges persist: 1) achieving high-quality translation remains a major hurdle, and 2) most existing methods heavily rely on large-scale parallel speech corpora, which are costly and difficult to obtain. To address these issues, we propose S2ST-Omni, an efficient and scalable framework for multilingual S2ST. Specifically, we decompose the S2ST task into speech-to-text translation (S2TT) and text-to-speech synthesis (TTS). For S2TT, we propose an effective speech language model that integrates the pretrained Whisper encoder for robust audio understanding and Qwen 3.0 for advanced text comprehension. A lightweight speech adapter is employed to bridge the modality gap between speech and text representations. To further facilitate multimodal knowledge learning, a two-stage fine-tuning strategy is introduced. In the TTS stage, we adopt a streaming autoregressive generation approach to produce natural and fluent target speech. Experiments on the CVSS benchmark show that S2ST-Omni consistently outperforms existing state-of-the-art S2ST systems in translation quality, highlighting its effectiveness and superiority.
Introduction
In the rapidly evolving landscape of AI-driven communication, Speech-to-Speech Translation (S2ST) remains a significant challenge. It requires a model to not only understand spoken nuances across diverse languages but also to translate that meaning accurately and synthesize it into natural-sounding speech. Current S2ST systems generally fall into two categories: End-to-End (E2E) models, which map source speech directly to target speech but are often data-hungry and difficult to debug, and Compositional models, which split the task into Speech-to-Text Translation (S2TT) and Text-to-Speech (TTS).
S2ST-Omni addresses the limitations of existing compositional models—which often fail to capture deep semantic relationships—by rethinking how Large Language Models (LLMs) are adapted for speech tasks. It introduces a framework that bridges a Whisper speech encoder with a Qwen 3.0 LLM using a novel "local-then-global" hybrid adapter and hierarchical language-aware features.
Methodology
The S2ST-Omni framework is built upon three innovative pillars designed to improve representation and translation accuracy:
- The Hybrid Speech Adapter: Instead of a simple linear projection, S2ST-Omni uses a "local-then-global" hierarchy. It first employs depthwise separable convolutions to capture local phonetic patterns, then downsamples the data to optimize compute, and finally applies Transformer blocks to model long-range semantic dependencies. This creates a cleaner abstraction for the LLM to process.
- Hierarchical Language-Aware (HLA) Augmentation: To address "language-blindness," the model injects source-language information at two levels:
- Acoustic Level (LA-Dual-CTC): Connectionist Temporal Classification (CTC) heads are conditioned on the source language using FiLM-style feature modulation, guiding acoustic layers with language-specific biases.
- Linguistic Level (LAP): The system dynamically generates prompts for the LLM (e.g., "The following is French speech. Translate it accurately into English.") to activate its internal multilingual knowledge.
- Progressive Fine-Tuning (PFT): To ensure stability when training a massive LLM with a new adapter, a two-stage approach is used. Stage I freezes the LLM and Encoder to train only the adapter for modality alignment. Stage II introduces LoRA (Low-Rank Adaptation) to the LLM to refine translation quality.
Experimental Results
The model was evaluated on the CVSS-C benchmark, focusing on three representative directions: French, Spanish, and German to English. S2ST-Omni set new benchmarks, significantly outperforming existing state-of-the-art systems.
As shown in the results, S2ST-Omni outperformed the strongest baseline (StreamSpeech) by an average of +24% BLEU and +31% ASR-BLEU. Notably, in the challenging German-to-English (De→En) direction, S2ST-Omni achieved 33.34 BLEU, a massive relative gain over previous methods. Ablation studies confirmed that both the acoustic-level (CTC) and linguistic-level (Prompting) language awareness are complementary and essential for peak performance.
Conclusion
S2ST-Omni represents a significant shift toward "SpeechLLMs" that truly understand the structural nuances of the languages they translate. By combining robust audio understanding with advanced text comprehension and a modular TTS backend, the framework provides a scalable solution for high-quality multilingual speech translation.
Summary updated