Abstract
We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following, without requiring task-specific audio instruction-tuning. Recent LALMs typically augment Large Language Models (LLMs) with auditory capabilities by training on large-scale, manually curated or LLM-synthesized audio-instruction datasets. However, these approaches have often suffered from the catastrophic forgetting of the LLM's original language abilities. To address this, we revisit the data construction pipeline and propose DeSTA, a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets. This approach preserves the LLM's native language proficiency while establishing effective audio-text alignment, thereby enabling zero-shot generalization without task-specific tuning. Using DeSTA, we construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench.
Introduction
Integrating auditory perception into Large Language Models (LLMs) typically involves cross-modal alignment, where audio features are mapped to a text-based embedding space. Standard practices involve fine-tuning an LLM on audio-description pairs generated by humans or teacher models like GPT-4. However, this often introduces a distribution mismatch. Forcing a model like Llama 3 to adopt the stylistic nuances of an external teacher can lead to "catastrophic forgetting," where the model loses its native reasoning and instruction-following capabilities.
DeSTA2.5-Audio addresses this by allowing the model to "learn in its own voice." By using a self-generated alignment strategy, the model gains auditory capabilities while maintaining the integrity of its original linguistic reasoning skills.
Methodology: The DeSTA Strategy
The core innovation of this work is the Descriptive Speech-Text Alignment (DeSTA) strategy. Instead of relying on external "gold-standard" labels, the researchers leverage the backbone LLM to generate its own training targets.
The process consists of two main phases:
- Dataset Construction: Metadata from 50 diverse audio datasets (covering speech, music, and environmental sounds) is converted into structured text. The backbone LLM (Llama 3.1-8B-Instruct) is then prompted to describe the audio based on this metadata. These self-generated descriptions serve as the training targets.
- Model Training: The architecture utilizes a frozen Whisper-large-v3 audio encoder and a frozen Llama 3.1 backbone. The only trainable component is a six-layer Q-Former modality adapter, which learns to align audio features with the LLM's self-generated text anchors.
This approach ensures that the training targets are perfectly aligned with the LLM's internal distribution, minimizing the risk of stylistic mismatch and forgetting.
Experimental Results
The model was trained on DeSTA-AQA5M, a dataset of 5 million triplets derived from 7,000 hours of audio. Despite using significantly less data than competitors (e.g., Qwen2-Audio-Instruct used 510,000 hours), DeSTA2.5-Audio achieved superior results across several benchmarks.
As shown in the table above, DeSTA2.5-Audio sets new state-of-the-art (SOTA) records on several key metrics:
- Dynamic-SUPERB Phase-1: Achieving an average score of 69.53, significantly outperforming previous models.
- Speech-IFEval: Maintaining an instruction-following rate (IFrate) of 93.89, demonstrating that the model retains its core linguistic capabilities after audio integration.
- SAKURA-Multi: Scoring 69.85, highlighting its strength in multi-hop reasoning tasks that require combining auditory input with external knowledge.
Discussion and Conclusion
The success of DeSTA2.5-Audio highlights the importance of self-consistency in cross-modal alignment. Training on self-generated text proves more effective than training on higher-quality but stylistically mismatched external data. Furthermore, the task-agnostic nature of the training allows the model to handle diverse auditory tasks zero-shot, without specific fine-tuning.
While the model is currently limited by the "text bridge"—meaning it can only understand what can be explicitly described in words—it represents a significant step toward robust, general-purpose Large Audio Language Models that do not sacrifice their native intelligence for new sensory capabilities.
Summary updated