Abstract
The auditory system plays a substantial role in shaping the overall human perceptual experience. While prevailing large language models (LLMs) and visual language models (VLMs) have shown their promise in solving a wide variety of language and vision understanding tasks, only a few of them can be generalised to the audio domain without compromising their domain-specific capability. In this work, we introduce Acoustic Prompt Tuning (APT), a new adapter extending LLMs and VLMs to the audio domain by injecting audio embeddings to the input of LLMs, namely soft prompting. Specifically, APT applies an instruction-aware audio aligner to generate soft prompts, conditioned on both input text and sounds, as the inputs to the language model. To mitigate data scarcity in the audio domain, a curriculum learning strategy is proposed by formulating diverse audio tasks in a sequential manner. Moreover, we improve the audio language model by using interleaved audio-text embeddings as the input sequence. In this improved model, zero constraints are imposed on the input format, thus it is capable of tackling diverse modelling tasks, such as few-shot audio classification and audio comparison. To further evaluate the advanced ability of the audio networks, we introduce natural language audio reasoning (NLAR), a new task that analyses two audio clips by comparison and summarisation. Experiments show that APT-enhanced LLMs (namely APT-LLMs) achieve competitive results compared to the expert models across various tasks. We finally demonstrate APT's ability in extending frozen VLMs to the audio domain without fine-tuning, achieving promising results in audio-visual question and answering.
Introduction
While Large Language Models (LLMs) and Vision-Language Models (VLMs) have achieved remarkable success in text and image processing, the audio domain remains a challenging frontier. Most existing audio-capable models are restricted by rigid input formats, typically limited to a single [audio, question, answer] structure. This rigidity prevents them from reasoning about multiple audio recordings simultaneously or performing "in-context learning"—the ability to learn from a few examples provided directly in the prompt.
Furthermore, the relative scarcity of high-quality audio-text data compared to text or image datasets poses a significant hurdle. Acoustic Prompt Tuning (APT) addresses these gaps by treating audio as a sequence of tokens that can be interleaved with text, facilitating natural, multi-turn, and multi-clip interactions.
Methodology
The core innovation is Acoustic Prompt Tuning (APT), a lightweight adapter that translates audio features into "soft prompts" that a frozen LLM can interpret.
Architecture
The APT-LLM system comprises three primary components:
- Audio Encoder: A pre-trained Audio-MAE extracts fine-grained acoustic patterns from input spectrograms.
- Audio Aligner: A Query-transformer (Q-former) style module uses cross-attention to "query" audio features based on text instructions. This ensures the generated soft prompts are task-relevant (e.g., focusing on specific sounds mentioned in the query).
- Frozen LLM: The resulting soft prompts are interleaved with standard text embeddings and processed by a model such as Vicuna.
Curriculum Learning
To overcome data scarcity, the model undergoes a three-stage curriculum training process:
- Stage 0: Initial audio-text alignment.
- Stage 1: Learning from single audio clips via tasks like tagging, captioning, and event detection.
- Stage 2: Advanced learning from multiple clips, enabling few-shot classification and complex audio comparison.
Evaluation and Benchmarking
The model was evaluated using several large-scale datasets, including AudioSet, WavCaps, AudioCaps, and Clotho. A significant contribution of this work is the introduction of the Natural Language Audio Reasoning (NLAR) benchmark. NLAR consists of 1,722 training and 1,640 test pairs requiring the model to compare or summarize two distinct audio clips (e.g., "Which recording has a more sudden sound?").
Key Insights
Interleaving and In-Context Learning
By allowing audio and text tokens to alternate freely, APT-LLM can perform in-context learning. It can "listen" to a few labeled examples and then classify a novel sound, mirroring the few-shot capabilities of text-based LLMs.
Instruction-Awareness
The APT aligner is not static; it uses the text prompt to filter audio information. This allows the model to focus on different acoustic features—such as background noise versus primary speakers—depending on the user's question.
Tri-modal Extension
APT can extend frozen VLMs (like BLIP-2) to the audio domain. By integrating APT into a model that already understands images and text, the researchers created a tri-modal model (APT-BLIP-2) capable of Audio-Visual Question Answering (AVQA) without vision-specific fine-tuning.
Results
APT-LLM achieves competitive performance against "expert" models specifically trained for audio tagging and captioning. On the NLAR reasoning task, APT-LLM achieved 62.9% accuracy, significantly outperforming a cascaded baseline (Captioning model + ChatGPT) at 27.9%. In 5-way 5-shot classification on ESC-50, it reached 91.0% accuracy. Furthermore, the tri-modal APT-BLIP-2 reached 59.7% accuracy in audio-visual tasks, demonstrating the synergy of combining modalities.
Conclusion and Future Work
While APT shows great promise, it remains dependent on the specific LLM embeddings used during training. Future work will explore expanding the model's scope to specialized domains like music theory and multi-talker speech transcription, as well as investigating the benefits of fine-tuning the LLM itself on audio-specific instructions.
Summary updated