Large Language Models (LLMs) possess a profound understanding of semantics, logic, and world knowledge, yet this "intelligence" is often trapped within the modality of text. When we adapt these models to process speech, they frequently suffer from a text-speech understanding gap: the model might be able to solve a complex logic puzzle in text but fail to do so when the same puzzle is spoken.
The research direction emerging across these 86 papers focuses on cross-modal knowledge distillation, a strategy that uses pre-trained text models as "teachers" to guide speech "students." By forcing speech models to mimic the internal representations or output behaviors of text models, researchers are bridging the gap between raw acoustic signals and high-level linguistic meaning.
The Asymmetry of Information
The fundamental problem in speech processing is that acoustic signals are high-dimensional, noisy, and redundant. In contrast, text is a highly compressed, discrete representation of meaning. As Santiago Cuervo and colleagues note in their work on the SALAD framework, text-based LLMs have already done the hard work of learning reasoning and world knowledge from massive corpora.
Cross-modal distillation works because it treats the text model as a semantic anchor. Instead of training a speech model from scratch to find meaning in waveforms, we tell the model: "Your goal is to produce a representation that looks exactly like what the text model produces when it reads the transcript of this audio."
Aligning the Latent Space
The most direct way to transfer knowledge is to align the internal "thoughts" of the two models. This is often achieved by minimizing the distance (typically via Mean Squared Error) between the hidden states of a speech encoder and a text encoder.
Kwanghee Choi and Hyung-Min Park demonstrated this with Distill-L2S, where a speech model is penalized if its feature vectors deviate from the semantic representations of a text-only teacher. This is particularly powerful for low-resource languages. While we may not have millions of hours of transcribed audio for a rare dialect, we often have plenty of text. Distillation allows the speech model to "inherit" the syntactic and semantic structures of that text.
Similarly, Linhao Dong and the team behind CIF-PT used a Continuous Integrate-and-Fire mechanism to align frame-level audio to token-level text, distilling knowledge from BERT to improve intent classification. By aligning these spaces, the speech encoder learns to ignore acoustic noise and focus on the "text-like" semantic core of the signal.
Mimicking Logic through Soft Labels
Beyond internal representations, distillation can happen at the output layer. When a text model predicts the next word in a sentence, it doesn't just pick one; it generates a probability distribution over all possible words (soft labels). These distributions contain rich information about which words are semantically related.
In the context of speech translation, Hao Xiong and Jiajun Zhang showed that training a student speech-to-text model to replicate the output probabilities of a high-performing text-to-text translation model significantly boosts accuracy. This "behavior imitation" is also central to the MTBI approach by Jingran Xie, which allows speech-capable LLMs to inherit the instruction-following capabilities of their text-only counterparts.
Overcoming the Data Bottleneck
A recurring theme in this research is efficiency. Traditional end-to-end speech models require massive amounts of paired (audio, transcript) data. Cross-modal distillation provides a shortcut.
Won Ik Cho and Nam Soo Kim found that distilling knowledge from BERT into a Spoken Language Understanding (SLU) system reduced error rates by 15%, even when labeled speech data was scarce. By leveraging the "semantic common sense" of the text model, the speech model doesn't have to learn what a "verb" or an "intent" is from scratch; it just has to learn how to map sounds to those pre-existing concepts.
This efficiency is pushed to the limit in SALAD, where Santiago Cuervo achieves state-of-the-art reasoning in speech models using 10x less training data than previous methods.
Multi-Stage and Hierarchical Guidance
The most sophisticated systems don't just distill once. They use hierarchical or multi-stage approaches to ensure the student model captures both broad meaning and fine-grained detail.
- Two-Stage Distillation: Gyuwan Kim and colleagues proposed a framework that aligns the speech encoder with a text teacher during both pre-training (for general semantics) and fine-tuning (for task-specific logic).
- Hierarchical Units: Joon-Hyuk Chang introduced a strategy that distills knowledge across different linguistic units—senones, phones, and subwords—ensuring the acoustic model understands the hierarchy of language.
- The Oracle Teacher: Ji Won Yoon developed the "Oracle Teacher," which is allowed to "peek" at ground-truth labels during training to provide even cleaner guidance to the student, fusing acoustic and linguistic context into a single supervisory signal.
Beyond Recognition: Translation and Synthesis
This paradigm extends beyond simple speech-to-text. In speech translation, Marco Gaido has demonstrated that word-level distillation from machine translation teachers helps end-to-end models overcome the inherent data scarcity of parallel speech-translation corpora.
Even in Text-to-Speech (TTS), the roles can be reversed to improve quality. Shun Lei and Zhiyong Wu use distillation to bridge the gap between audio-based style extraction and text-based style prediction. By training a text-only predictor to mimic style embeddings from an audio teacher, they enable models to generate highly expressive prosody from text alone, without needing a reference audio clip at inference time.
Conclusion
The "text-speech gap" is being closed not just by collecting more audio data, but by smarter knowledge transfer. By treating text models as repositories of linguistic logic, researchers like Santiago Cuervo, Marco Gaido, and Won Ik Cho are creating speech systems that don't just "hear" words, but "understand" the underlying meaning with the same depth as our most advanced text-based AI.