Instituto de Telecomunicações at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning

Giuseppe Attanasio, Sonal Sannigrahi, Ben Peters, André F. T. Martins

TL;DR

This paper introduces an efficient, small-scale speech-to-text model (<2B parameters) developed for the IWSLT 2025 instruction-following shared task. By aligning a pretrained w2v-BERT 2.0 speech encoder with a Qwen 2.5 text decoder using a two-stage training curriculum, the authors demonstrate how to build a lightweight, open-licensed multimodal model capable of speech recognition, translation, and question answering.

What Problem Does This Paper Address?

Most state-of-the-art multimodal models that process speech rely on massive text language model backbones (typically 7B parameters or larger). While powerful, these models are computationally expensive to train, deploy, and iterate upon.

The authors from Instituto de Telecomunicações address this efficiency gap by exploring the limits of Small Language Models (SLMs) under the 2B parameter threshold. They investigate whether a highly optimized training curriculum combined with strictly open-licensed (CC-BY) and high-quality synthetic data can enable small-scale models to handle complex, multi-task speech instructions across English, German, and Chinese.

Methods

The authors implement a unified speech-to-text architecture and train it using a distinct two-stage curriculum:

  1. Model Architecture: The speech stack extracts audio features, compresses them via 1D convolutional layers, and processes them through a pretrained w2v-BERT 2.0 encoder. A modality and length adapter (consisting of two Conformer-like layers) then projects these speech representations directly into the embedding space of the text decoder (Qwen 2.5 1.5B).
  2. Modality Alignment (Stage 1): To teach the text model how to interpret speech embeddings, the authors freeze the text decoder and train only the convolutional layers and the modality adapter on Automatic Speech Recognition (ASR) data. They introduce a "length hint" token during this phase to help the model learn the underlying audio-to-text length distribution.
  3. Instruction Fine-Tuning (Stage 2): In this phase, all components are trained jointly end-to-end. The model is fed a mixture of ASR, Speech Translation (ST), and Spoken Question Answering (SQA) tasks. Generation is explicitly guided by prepending target language (<LANG>) and task (<TASK>) tokens to the text context.
Figure 1: Illustration of our model. During training, the speech stack (red) generates speech representations which are prepended to task and language tags and an (optional) length hint (yellow), and text tokens (green). At inference time, we only provide the language and task tokens.

Datasets

The authors rely exclusively on open-licensed datasets and synthetic data generation:

  • Modality Alignment: LibriSpeech, Multilingual LibriSpeech, FLEURS, and CommonVoice 16.1 (all under CC-BY 4.0).
  • Instruction Fine-Tuning: The alignment datasets plus VoxPopuli, The People's Speech, CoVoST-2, and SpokenSQuAD.
  • Synthetic Data Generation: To overcome data scarcity in translation and question answering, the authors used an ensemble of machine translation models (TowerInstruct, EuroLLM, NLLB) filtered by a COMETKiwi quality estimation oracle to pseudolabel English speech data into German and Chinese. They also used Qwen2.5-70B to generate synthetic, unanswerable questions to ensure the SQA dataset remained balanced.

Key Insights and Ideas

  • ASR Data Dominance: A major hurdle in multi-task speech models is that ASR data tends to dominate the model's capabilities. Because ASR training is more straightforward, the small model frequently defaulted to transcribing audio even when explicitly instructed to translate it.
  • Curriculum Balance is Vital: For small-scale models (<2B parameters), simply mixing data isn't enough. The authors note that a more carefully balanced data distribution during the instruction fine-tuning phase is required to prevent the model from losing its multi-task capabilities.
  • Multi-Model Pseudolabeling Yields Better Data: Instead of relying on a single translation model for synthetic data, using an ensemble of four distinct translation models combined with an algorithmic quality filter (COMETKiwi score $\ge$ 0.85) significantly increased both data diversity and the total volume of usable training examples.

Key Results

The model's performance varied significantly across the three evaluation tasks:

  • ASR: The model achieved a strong Word Error Rate (WER) of 0.15 for English, showing robust performance even when dealing with spontaneous speech and high speaker variability.
  • Speech Translation (ST): The model scored a COMET score of 0.34 for both English-to-German and English-to-Chinese. However, manual inspection revealed qualitative issues, such as the model repeating words or ignoring the translation tag entirely.
  • Spoken Question Answering (SQA): The model struggled with this complex task, achieving a BertScore of 0.14 for English, 0.22 for German, and 0.21 for Chinese. At test time, the model occasionally generated the question itself rather than the answer.

Limitations and Future Work

The authors identify several clear bottlenecks in their current system:

  • Audio Length Cutoff: Due to hardware memory limitations, audio inputs were capped at 120 seconds. This forced the system to discard a significant portion of the longer SpokenSQuAD data, directly hurting its multi-task performance.
  • Capacity Constraints: The 1.5B parameter text backbone lacked the raw reasoning and multilingual capacity required to execute high-quality translation and contextual question answering simultaneously.

To address these limitations, future work will focus on scaling up to 3B parameter base models, implementing longer audio context support, and refining the synthetic data filtering pipeline for the question-answering domain.

Summary updated

Abstract

This paper presents the IT-IST submission to the IWSLT 2025 Shared Task on Instruction Following Speech Processing. We submit results for the Short Track, i.e., speech recognition, translation, and spoken question answering. Our model is a unified speech-to-text model that integrates a pre-trained continuous speech encoder and text decoder through a first phase of modality alignment and a second phase of instruction fine-tuning. Crucially, we focus on using small-scale language model backbones (< 2B) and restrict to high-quality, CC-BY data along with synthetic data generation to supplement existing resources.
BibTeX
@misc{attanasio2025instituto,
  author = {Giuseppe Attanasio and Sonal Sannigrahi and Ben Peters and André F. T. Martins},
  title = {{Instituto de Telecomunicações at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning}},
  year = {2025},
  eprint = {2506.17019},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.17019},
}
Figure