From Flat Language Labels to Typological Priors: Structured Language Conditioning for Multilingual Speech-to-Speech Translation

Yu Pan, Yang Hou, Xiongfei Wu, Liang Zhang, Yves Le Traon, Lei Ma, Jianjun ZhaoarXiv 2026 · arXiv

TL;DR

S2ST-Omni 2 is a novel speech-to-speech translation framework that replaces simple, flat language labels with structured linguistic "typological priors" (like word order, morphology, and language family). By injecting these structural rules into the model, it achieves state-of-the-art translation quality and shows massive data-efficiency gains, especially when training data is scarce.

What Problem Does This Paper Address?

Multilingual Speech-to-Speech Translation (S2ST) systems typically handle different source languages using a "language-as-label" approach. They tag an input with a flat identifier or embedding (like [FR] for French or [DE] for German) to tell the model what language it is processing.

However, this design completely ignores the rich, systematic structural relationships shared across languages. For instance, French and Spanish share a common ancestry (Romance family) and similar Subject-Verb-Object (SVO) word orders, while German often features clause-final verb placement. By treating every language as an isolated island, standard models waste parameters and data capacity relearning these basic structural rules from scratch. This lack of linguistic awareness makes it incredibly difficult to train high-quality S2ST models when supervised data is limited.

Methods

S2ST-Omni 2 addresses this by redesigning the language-conditioning pathway of a compositional SpeechLLM framework across three distinct levels:

Fig. 1. Overall architecture and two-stage training pipeline of S2ST-Omni 2. LA denotes language-aware, CE is cross-entropy, and src/tgt denote source/target. TI-HLE and Dynamically-Gated LA-Dual-CTC are training-time auxiliary modules, whereas typology-aware prompting is retained during inference.
  1. Representation Level (TI-HLE): The authors introduce Typology-Informed Hierarchical Language Encoding. Instead of a single flat vector, the source language is decomposed into four distinct feature channels: morphology (e.g., fusional vs. agglutinative), an English-directed reordering profile (e.g., SVO-oriented vs. verb-final), genealogical family (e.g., Romance, Germanic, Japonic), and a language-specific residual channel to capture fine-grained nuances not covered by the broad categories.
  2. Acoustic Level (Dynamically-Gated LA-Dual-CTC): To inject these typological features into the acoustic representation, the model uses Feature-wise Linear Modulation (FiLM). Crucially, instead of applying this modulation uniformly, a Dynamic Frame Gate adaptively scales the influence of the typological prior frame-by-frame based on the actual speech content.
  3. Decoding Level (Typology-Aware Prompting): Translation-oriented linguistic priors are explicitly written into the LLM decoder's system prompt. For example, the German prompt explicitly instructs the model to pay attention to compound word decomposition and clause-final reordering.

The framework uses a two-stage progressive fine-tuning strategy. The heavy speech encoder (Whisper) and the core Large Language Model (Qwen3) remain mostly frozen, while the typological modules and a low-rank adaptation (LoRA) layer are optimized.

Datasets

The model was evaluated on the CVSS-C corpus, a standard benchmark derived from CoVoST 2 that features multiple source languages paired with synthesized English target speech.

  • Main Multilingual Setting: Jointly trained on French (264 hours), German (184 hours), and Spanish (113 hours) translating into English, totaling 561 hours.
  • Low-Resource Extension: Evaluated on Japanese-to-English using a highly restricted data budget of only ~3 hours of supervised training data to test the model's robustness on a typologically distant language.

Key Insights and Ideas

  • Linguistic Structure as a Regularizer: Grouping languages by shared structural traits (like word order or morphology) acts as a powerful inductive bias. It allows the model to share parameters effectively across related languages rather than forcing each language pathway to operate independently.
  • The Scarcity Advantage: Typological priors act as a safety net when data is scarce. When a model has access to hundreds of hours of data, it can eventually infer structural rules implicitly. However, when data is limited, explicitly telling the model how a language behaves structurally prevents it from making catastrophic translation errors.
  • Content-Adaptive Modulation: Language characteristics shouldn't be forced onto speech features uniformly. The success of the dynamic frame gate proves that typological priors are highly relevant during certain acoustic segments (like complex verbs or structural transitions) and less relevant during others.

Key Results

S2ST-Omni 2 achieved the highest average performance across all evaluated metrics (BLEU, COMET, ASR-BLEU, and BLASER 2.0) compared to other representative S2ST systems.

  • Compared directly to its predecessor, S2ST-Omni, it achieved average relative gains of 5.8% in text BLEU and 4.6% in speech ASR-BLEU.
  • The largest gains were observed in German-to-English translation, confirming that explicit typological guidance is highly effective at resolving severe structural mismatches (like German's verb-final tendencies).
BLEU under varying training data budgets for S2ST-Omni and S2ST-Omni 2.
  • Data Efficiency: As shown in Figure 4, the relative BLEU advantage over the baseline increased monotonically as the training data budget shrank—leaping from a +5.8% relative gain at 561 hours to a massive +15.1% relative gain at just 30 hours.
  • Ultra-Low Resource Japanese: With only ~3 hours of training data, S2ST-Omni 2 outperformed the baseline by +2.39 BLEU and +2.34 ASR-BLEU, proving that these priors generalize well to typologically distant languages.

Limitations and Future Work

The authors acknowledge that their current typological assignments are necessarily coarse-grained and translation-oriented, rather than exhaustive linguistic classifications. Furthermore, the current framework is tailored to a many-to-one translation paradigm (multiple source languages into English).

Future work could explore expanding this approach to many-to-many translation setups and developing methods to automatically discover or refine typological features directly from data, reducing the reliance on manual linguistic mapping.

Summary updated

Abstract

Compositional speech-to-speech translation (S2ST) systems built upon speech large language models (SpeechLLMs) have recently shown promising performance. However, existing S2ST systems often either neglect source-language information or encode it through a language-as-label paradigm, representing each source language as an independent flat embedding. Such a design overlooks systematic linguistic structure shared across languages, which may limit data-efficient multilingual adaptation when supervised S2ST data are scarce. To address this issue, we propose S2ST-Omni 2, a many-to-one compositional S2ST framework that systematically reformulates multilingual language conditioning from flat language labels to structured typological priors. Specifically, S2ST-Omni 2 revisits language conditioning at three levels: typology-informed hierarchical language encoding for structured source-language representation, dynamically-gated language-aware Dual-CTC for content-adaptive acoustic modulation, and typology-aware LLM prompting for decoder-side linguistic guidance. Experiments on CVSS-C show that S2ST-Omni 2 achieves superior average performance among representative S2ST approaches across BLEU, COMET, ASR-BLEU, and BLASER 2.0 under the adopted evaluation protocol. Ablation studies indicate that the proposed representation-level, acoustic-level, and decoding-level strategies provide complementary benefits. Moreover, controlled data-budget analyses and a Japanese-to-English evaluation using only approximately 3 hours of supervised training data suggest that explicit typological priors provide useful inductive biases for data-efficient multilingual S2ST.
BibTeX
@misc{pan2026from,
  author = {Yu Pan and Yang Hou and Xiongfei Wu and Liang Zhang and Yves Le Traon and Lei Ma and Jianjun Zhao},
  title = {{From Flat Language Labels to Typological Priors: Structured Language Conditioning for Multilingual Speech-to-Speech Translation}},
  year = {2026},
  eprint = {2605.16026},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.16026},
}
Figure