TL;DR
This paper introduces DeSTA2.5-Audio, a novel Large Audio Language Model (LALM) that tackles the "catastrophic forgetting" problem in existing models. By using a self-generated cross-modal alignment strategy, DeSTA2.5-Audio preserves the LLM's core language abilities while achieving state-of-the-art performance across a wide range of audio understanding tasks, even with significantly less training data.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown remarkable capabilities in understanding and generating text. A natural extension is to equip them with auditory perception, leading to Large Audio Language Models (LALMs). These models typically combine a pre-trained audio model with an LLM, aligning them through cross-modal training on audio-text instruction datasets. However, a significant challenge arises: existing LALMs often suffer from "catastrophic forgetting," where they become overly specialized to audio tasks, losing their original text-based knowledge and general reasoning abilities. This happens because the training data, often generated by external LLMs or human annotators, introduces a distribution shift, forcing the model to learn new, foreign patterns at the expense of its native ones. Balancing robust auditory perception with the retention of core LLM capabilities is therefore a critical hurdle.
Methods
The core innovation is the DeSTA (Descriptive Speech-Text Alignment) framework, specifically DeSTA2.5-Audio. Instead of relying on external data sources for training targets, DeSTA leverages the backbone LLM itself to generate its own training targets.
Here's how it works:
- Structured Audio Descriptions: Metadata from audio clips (e.g., speaker gender, emotion, background sounds) is converted into a structured textual description.
- Self-Generated Targets: This description is paired with a randomly sampled text prompt (e.g., "Describe the audio," "What is the emotion?"). The backbone LLM then generates a response to this prompt based on the description. This LLM-generated response becomes the training target.
- Cross-Modal Alignment: The model is then trained on triplets of (audio input, prompt, self-generated target). This ensures that the training targets are stylistically and semantically consistent with the LLM's native output distribution, preventing the distribution shift that causes catastrophic forgetting.
- Modular Architecture: DeSTA2.5-Audio uses a modular architecture, integrating a pre-trained audio model with an instruction-tuned LLM. A modality adapter (Q-Former blocks) bridges the two. Crucially, the audio model and LLM parameters are frozen, and only the modality adapter is fine-tuned, further preserving the LLM's knowledge.
Datasets
The authors constructed DeSTA-AQA5M, a large-scale, task-agnostic dataset comprising approximately 5 million audio-prompt-response triplets. This dataset is derived from 7,000 hours of audio spanning 50 diverse publicly available datasets, covering speech, environmental sounds, and music. This broad coverage is key to achieving general-purpose capabilities.
For evaluation, they used several established benchmarks:
- Dynamic-SUPERB (Phase-1 & Phase-2): Evaluates instruction-following and speech understanding across a wide range of tasks.
- MMAU: Assesses advanced audio-language understanding and reasoning.
- SAKURA (Single & Multi): Tests single-hop and multi-hop reasoning, requiring world knowledge.
- Speech-IFEval: Specifically designed to measure instruction-following ability and quantify catastrophic forgetting.
- VoiceBench: Evaluates spoken interaction performance.
Key Insights and Ideas
- Self-Generated Data is Key to Avoiding Catastrophic Forgetting: The most significant insight is that having the LLM generate its own training targets is crucial for preserving its native language abilities. This self-consistency in training data directly combats the distribution shift that plagues other LALMs.
- Efficiency Through Data Quality: DeSTA2.5-Audio achieves state-of-the-art results with only 7,000 hours of training data, vastly outperforming models like Qwen2-Audio-Instruct which used 510,000 hours. This highlights that the quality and consistency of training data can be more impactful than sheer quantity.
- Robustness to Distribution Mismatch: The paper demonstrates that even using a more powerful LLM (like Qwen2.5) to generate targets for a weaker LLM (like Llama3.1) leads to severe performance degradation if the data isn't self-generated. This underscores the importance of the self-generation mechanism itself.
- Deferred Cross-Modal Integration: Representation-level analysis reveals that audio information isn't immediately integrated in the early layers of the LLM. Instead, it's progressively incorporated into the LLM's semantic space in deeper layers, suggesting a more nuanced and deferred integration process.
- Uncertainty Awareness is Valuable: DeSTA2.5-Audio sometimes responds with "I don't have enough information" on unfamiliar tasks, demonstrating valuable uncertainty awareness for real-world deployment.
Key Results
- State-of-the-Art Performance: DeSTA2.5-Audio achieved state-of-the-art or competitive results across multiple benchmarks, including Dynamic-SUPERB (69.53 avg), MMAU (57.50), SAKURA-Multi (69.85), and Speech-IFEval (93.89 IFrate).
- Superior Instruction-Following: It significantly outperformed other models on SAKURA-Multi and Speech-IFEval, demonstrating strong multi-hop reasoning and retention of instruction-following capabilities.
- Efficiency: Achieved top performance with only 7,000 hours of training data, compared to over 510,000 hours for a strong baseline.
- Robustness: Consistently outperformed models trained on external, mismatched data sources, even when those sources were generated by more powerful LLMs.
- VoiceBench Performance: Achieved a strong overall score of 74.52 on VoiceBench, outperforming many other fusion-based models and demonstrating good conversational abilities.
Limitations and Future Work
- Textual Description Limitations: The authors acknowledge that not all acoustic nuances can be perfectly captured through textual descriptions.
- Future Directions: They propose exploring audio-to-audio supervision and incorporating human feedback (preference learning) to better capture subtle, non-text-expressible acoustic features, aiming for LALMs with more complete and human-like audio perception.
Summary updated