Self-Powered LLM Modality Expansion for Large Speech-Text Models

Tengfei Yu, Xuebo Liu, Zhiyi Hou, Liang Ding, Dacheng Tao, Min Zhang

Abstract

Large Speech-Text Models (LSMs) aim to integrate auditory processing into the reasoning frameworks of Large Language Models (LLMs). However, current training paradigms often lead to "speech anchor bias," where models default to simple transcription and ignore nuanced textual instructions. We introduce a self-powered expansion framework that utilizes the LLM’s internal text-processing intelligence to bootstrap diverse speech-instruction data. By pairing ASR transcripts with LLM-generated tasks, we significantly improve the model's instruction-following capabilities across multiple speech-language understanding benchmarks.

Introduction: The Speech Anchor Bias

The primary challenge in developing effective LSMs is ensuring the model treats speech as a flexible input modality rather than a fixed trigger for transcription. Most LSMs are fine-tuned on Automatic Speech Recognition (ASR) datasets, which primarily consist of "Speech -> Transcript" pairs. This creates a "speech anchor bias": the model learns to prioritize the speech signal so heavily that it ignores the accompanying text prompt.

Comparison between successful instruction following (left) and speech anchor bias (right) where the model defaults to transcription.

As illustrated above, a vanilla LSM might respond to a prompt like "Summarize this speech" by providing a verbatim transcript instead of a summary. This failure to integrate instructions prevents LLMs from truly incorporating speech into their reasoning framework.

Methodology: The Self-Powered Framework

To overcome this bias, we propose a modular architecture that bridges the gap between speech and text through a "self-powered" data generation loop.

Architecture

The system consists of three main components: a Speech Encoder (Whisper-small), a Connector (Q-Former), and a Backbone LLM (Vicuna-7B-1.5). The Q-Former maps variable-length acoustic features into a fixed set of textual tokens compatible with the LLM's embedding space.

The modular architecture of the Self-Powered LSM, showing the flow from speech input and text instruction to the final assistant response.

Self-Powered Data Generation

The innovation lies in leveraging the LLM's existing text-based proficiency to generate training data. We take standard ASR transcripts and prompt the LLM (in text-only mode) to perform various tasks—such as keyword extraction, sentiment analysis, or translation. We then pair the original speech audio with these self-generated instructions and targets. This process "distills" the LLM's text-domain intelligence into the speech modality, forcing the model to look at the instruction to determine its response.

Key Results and Insights

The Self-Powered LSM demonstrates superior performance across both traditional ASR and complex Speech-Language Understanding (SLU) tasks.

  • Enhanced ASR Accuracy: On the LibriSpeech (clean) benchmark, the model achieved a Word Error Rate (WER) of 3.8, a significant improvement over the 7.8 WER of vanilla instruction-tuned models.
  • Multi-task Versatility: Unlike vanilla models that fail at non-transcription tasks, the Self-Powered LSM achieved competitive scores on MELD (51.4 F1 for Emotion Recognition) and FSC (47.2 accuracy for Intent Classification).
  • Instruction Salience: Layer-wise attention analysis confirmed that our method encourages deeper transformer layers to attend to the instruction tokens rather than just the speech tokens, effectively mitigating the speech anchor bias.

Conclusion

By using an LLM to bootstrap its own multimodal training data, we can effectively expand its capabilities into the speech domain without the need for massive, manually annotated datasets. This "self-powered" approach ensures that the model maintains its reasoning flexibility, treating speech as a first-class citizen alongside text. Future work will focus on refining the quality of self-generated data to further close the performance gap with cascaded speech-to-text systems.

Summary updated

BibTeX
@inproceedings{yu2024self,
  author = {Tengfei Yu and Xuebo Liu and Zhiyi Hou and Liang Ding and Dacheng Tao and Min Zhang},
  title = {{Self-Powered LLM Modality Expansion for Large Speech-Text Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.690},
}
Figure