TL;DR
This paper introduces an efficient, small-scale speech-to-text model (<2B parameters) developed for the IWSLT 2025 instruction-following shared task. By aligning a pretrained w2v-BERT 2.0 speech encoder with a Qwen 2.5 text decoder using a two-stage training curriculum, the authors demonstrate how to build a lightweight, open-licensed multimodal model capable of speech recognition, translation, and question answering.
What Problem Does This Paper Address?
Most state-of-the-art multimodal models that process speech rely on massive text language model backbones (typically 7B parameters or larger). While powerful, these models are computationally expensive to train, deploy, and iterate upon.
The authors from Instituto de Telecomunicações address this efficiency gap by exploring the limits of Small Language Models (SLMs) under the 2B parameter threshold. They investigate whether a highly optimized training curriculum combined with strictly open-licensed (CC-BY) and high-quality synthetic data can enable small-scale models to handle complex, multi-task speech instructions across English, German, and Chinese.
Methods
The authors implement a unified speech-to-text architecture and train it using a distinct two-stage curriculum:
- Model Architecture: The speech stack extracts audio features, compresses them via 1D convolutional layers, and processes them through a pretrained
w2v-BERT 2.0encoder. A modality and length adapter (consisting of two Conformer-like layers) then projects these speech representations directly into the embedding space of the text decoder (Qwen 2.5 1.5B). - Modality Alignment (Stage 1): To teach the text model how to interpret speech embeddings, the authors freeze the text decoder and train only the convolutional layers and the modality adapter on Automatic Speech Recognition (ASR) data. They introduce a "length hint" token during this phase to help the model learn the underlying audio-to-text length distribution.
- Instruction Fine-Tuning (Stage 2): In this phase, all components are trained jointly end-to-end. The model is fed a mixture of ASR, Speech Translation (ST), and Spoken Question Answering (SQA) tasks. Generation is explicitly guided by prepending target language (
<LANG>) and task (<TASK>) tokens to the text context.
Datasets
The authors rely exclusively on open-licensed datasets and synthetic data generation:
- Modality Alignment: LibriSpeech, Multilingual LibriSpeech, FLEURS, and CommonVoice 16.1 (all under CC-BY 4.0).
- Instruction Fine-Tuning: The alignment datasets plus VoxPopuli, The People's Speech, CoVoST-2, and SpokenSQuAD.
- Synthetic Data Generation: To overcome data scarcity in translation and question answering, the authors used an ensemble of machine translation models (TowerInstruct, EuroLLM, NLLB) filtered by a
COMETKiwiquality estimation oracle to pseudolabel English speech data into German and Chinese. They also usedQwen2.5-70Bto generate synthetic, unanswerable questions to ensure the SQA dataset remained balanced.
Key Insights and Ideas
- ASR Data Dominance: A major hurdle in multi-task speech models is that ASR data tends to dominate the model's capabilities. Because ASR training is more straightforward, the small model frequently defaulted to transcribing audio even when explicitly instructed to translate it.
- Curriculum Balance is Vital: For small-scale models (<2B parameters), simply mixing data isn't enough. The authors note that a more carefully balanced data distribution during the instruction fine-tuning phase is required to prevent the model from losing its multi-task capabilities.
- Multi-Model Pseudolabeling Yields Better Data: Instead of relying on a single translation model for synthetic data, using an ensemble of four distinct translation models combined with an algorithmic quality filter (
COMETKiwiscore $\ge$ 0.85) significantly increased both data diversity and the total volume of usable training examples.
Key Results
The model's performance varied significantly across the three evaluation tasks:
- ASR: The model achieved a strong Word Error Rate (WER) of 0.15 for English, showing robust performance even when dealing with spontaneous speech and high speaker variability.
- Speech Translation (ST): The model scored a COMET score of 0.34 for both English-to-German and English-to-Chinese. However, manual inspection revealed qualitative issues, such as the model repeating words or ignoring the translation tag entirely.
- Spoken Question Answering (SQA): The model struggled with this complex task, achieving a BertScore of 0.14 for English, 0.22 for German, and 0.21 for Chinese. At test time, the model occasionally generated the question itself rather than the answer.
Limitations and Future Work
The authors identify several clear bottlenecks in their current system:
- Audio Length Cutoff: Due to hardware memory limitations, audio inputs were capped at 120 seconds. This forced the system to discard a significant portion of the longer SpokenSQuAD data, directly hurting its multi-task performance.
- Capacity Constraints: The 1.5B parameter text backbone lacked the raw reasoning and multilingual capacity required to execute high-quality translation and contextual question answering simultaneously.
To address these limitations, future work will focus on scaling up to 3B parameter base models, implementing longer audio context support, and refining the synthetic data filtering pipeline for the question-answering domain.
Summary updated