DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment

Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, Yi-Cheng Lin, Yu-Xiang Lin, Chi-An Fu, Chun-Yi Kuan, Wenze Ren, Xuanjun Chen, Wei-Ping Huang, En-Pei Hu, Tzu-Quan Lin, Yuan-Kuei Wu, Kuan-Po Huang, Hsiao-Ying Huang, Huang-Cheng Chou, Kai-Wei Chang, Cheng-Han Chiang, Boris Ginsburg, Yu-Chiang Frank Wang, Hung-yi Lee

Abstract

We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following, without requiring task-specific audio instruction-tuning. Recent LALMs typically augment Large Language Models (LLMs) with auditory capabilities by training on large-scale, manually curated or LLM-synthesized audio-instruction datasets. However, these approaches have often suffered from the catastrophic forgetting of the LLM's original language abilities. To address this, we revisit the data construction pipeline and propose DeSTA, a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets. This approach preserves the LLM's native language proficiency while establishing effective audio-text alignment, thereby enabling zero-shot generalization without task-specific tuning. Using DeSTA, we construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench.

Introduction

Integrating auditory perception into Large Language Models (LLMs) typically involves cross-modal alignment, where audio features are mapped to a text-based embedding space. Standard practices involve fine-tuning an LLM on audio-description pairs generated by humans or teacher models like GPT-4. However, this often introduces a distribution mismatch. Forcing a model like Llama 3 to adopt the stylistic nuances of an external teacher can lead to "catastrophic forgetting," where the model loses its native reasoning and instruction-following capabilities.

DeSTA2.5-Audio addresses this by allowing the model to "learn in its own voice." By using a self-generated alignment strategy, the model gains auditory capabilities while maintaining the integrity of its original linguistic reasoning skills.

Methodology: The DeSTA Strategy

The core innovation of this work is the Descriptive Speech-Text Alignment (DeSTA) strategy. Instead of relying on external "gold-standard" labels, the researchers leverage the backbone LLM to generate its own training targets.

Overview of the DeSTA strategy, showing dataset construction via self-generation and the subsequent model training phase.

The process consists of two main phases:

  1. Dataset Construction: Metadata from 50 diverse audio datasets (covering speech, music, and environmental sounds) is converted into structured text. The backbone LLM (Llama 3.1-8B-Instruct) is then prompted to describe the audio based on this metadata. These self-generated descriptions serve as the training targets.
  2. Model Training: The architecture utilizes a frozen Whisper-large-v3 audio encoder and a frozen Llama 3.1 backbone. The only trainable component is a six-layer Q-Former modality adapter, which learns to align audio features with the LLM's self-generated text anchors.

This approach ensures that the training targets are perfectly aligned with the LLM's internal distribution, minimizing the risk of stylistic mismatch and forgetting.

Experimental Results

The model was trained on DeSTA-AQA5M, a dataset of 5 million triplets derived from 7,000 hours of audio. Despite using significantly less data than competitors (e.g., Qwen2-Audio-Instruct used 510,000 hours), DeSTA2.5-Audio achieved superior results across several benchmarks.

Performance comparison of DeSTA2.5-Audio against state-of-the-art models across multiple audio-language benchmarks.

As shown in the table above, DeSTA2.5-Audio sets new state-of-the-art (SOTA) records on several key metrics:

  • Dynamic-SUPERB Phase-1: Achieving an average score of 69.53, significantly outperforming previous models.
  • Speech-IFEval: Maintaining an instruction-following rate (IFrate) of 93.89, demonstrating that the model retains its core linguistic capabilities after audio integration.
  • SAKURA-Multi: Scoring 69.85, highlighting its strength in multi-hop reasoning tasks that require combining auditory input with external knowledge.

Discussion and Conclusion

The success of DeSTA2.5-Audio highlights the importance of self-consistency in cross-modal alignment. Training on self-generated text proves more effective than training on higher-quality but stylistically mismatched external data. Furthermore, the task-agnostic nature of the training allows the model to handle diverse auditory tasks zero-shot, without specific fine-tuning.

While the model is currently limited by the "text bridge"—meaning it can only understand what can be explicitly described in words—it represents a significant step toward robust, general-purpose Large Audio Language Models that do not sacrifice their native intelligence for new sensory capabilities.

Summary updated

Abstract

We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following, without requiring task-specific audio instruction-tuning. Recent LALMs typically augment Large Language Models (LLMs) with auditory capabilities by training on large-scale, manually curated or LLM-synthesized audio-instruction datasets. However, these approaches have often suffered from the catastrophic forgetting of the LLM's original language abilities. To address this, we revisit the data construction pipeline and propose DeSTA, a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets. This approach preserves the LLM's native language proficiency while establishing effective audio-text alignment, thereby enabling zero-shot generalization without task-specific tuning. Using DeSTA, we construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench. Comprehensive comparative studies demonstrate that our self-generated strategy outperforms widely adopted data construction and training strategies in both auditory perception and instruction-following capabilities. Our findings underscore the importance of carefully designed data construction in LALM development and offer practical insights for building robust, general-purpose LALMs.
BibTeX
@article{lu2026desta,
  author = {Ke-Han Lu and Zhehuai Chen and Szu-Wei Fu and Chao-Han Huck Yang and Sung-Feng Huang and Chih-Kai Yang and Chee-En Yu and Chun-Wei Chen and Wei-Chih Chen and Chien-yu Huang and Yi-Cheng Lin and Yu-Xiang Lin and Chi-An Fu and Chun-Yi Kuan and Wenze Ren and Xuanjun Chen and Wei-Ping Huang and En-Pei Hu and Tzu-Quan Lin and Yuan-Kuei Wu and Kuan-Po Huang and Hsiao-Ying Huang and Huang-Cheng Chou and Kai-Wei Chang and Cheng-Han Chiang and Boris Ginsburg and Yu-Chiang Frank Wang and Hung-yi Lee},
  title = {{DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment}},
  journal = {IEEE Transactions on Audio Speech and Language Processing},
  year = {2026},
  doi = {10.1109/taslpro.2026.3675792},
}
Figure