Language-based Audio Retrieval with Co-Attention Networks

Haoran Sun, Zimu Wang, Qiuyi Chen, Jianjun Chen, Jia Wang, Haiyang Zhang

Abstract

Language-based audio retrieval aims to identify specific audio clips from a database using natural language queries. This task is complicated by the "heterogeneity gap" between continuous audio signals and discrete text. We present a framework utilizing cascaded co-attention networks to bridge this gap. By implementing deep stacking and iterating modules, the proposed approach facilitates bidirectional interaction between modalities. Experimental results on the Clotho and AudioCaps datasets demonstrate significant performance improvements, including a 16.6% increase in mean Average Precision (mAP) on Clotho, establishing a new state-of-the-art for the task.

1. Introduction

The proliferation of user-generated audio content has created a pressing need for efficient retrieval systems. Unlike traditional keyword searches, language-based audio retrieval allows users to use complex natural language descriptions, such as "the sound of a printer jam" or "birds chirping in a rainy forest."

The primary obstacle in this field is the semantic alignment of disparate data types. Audio signals are high-dimensional, temporal sequences, whereas text queries are short and symbolic. Existing models often rely on independent self-attention mechanisms for each modality before merging them into a common vector space. However, these methods frequently fail to capture the fine-grained, cross-modal dependencies necessary for understanding complex acoustic scenes.

2. Methodology

The proposed framework addresses the heterogeneity gap through three primary innovations: advanced encoding with data augmentation, a bidirectional co-attention mechanism, and cascaded architectural configurations.

2.1 Encoders and GPTtar Augmentation

The system utilizes CLAP (Contrastive Language-Audio Pretraining) for audio feature extraction and RoBERTa for text encoding. To mitigate the scarcity of diverse human-written captions, the authors introduced GPTtar. This technique uses ChatGPT to generate five semantically equivalent variations for each original caption, effectively expanding the linguistic diversity of the training data.

2.2 Co-Attention Mechanism

At the heart of the framework is the co-attention module. Unlike standard self-attention, which operates within a single modality, co-attention allows audio features to serve as "queries" for text "keys/values," and vice versa. This bidirectional flow ensures that the model identifies specific linguistic tokens that correspond to distinct temporal segments in the audio.

2.3 Cascaded Architectures

To explore the benefits of deep cross-modal reasoning, two cascaded configurations were evaluated:

  • Stacking Module: This architecture layers multiple co-attention units sequentially, allowing for progressively deeper feature integration.
  • Iterating Module: This hierarchical approach first refines the text representation and subsequently uses this refined information to re-query the audio segments. This iterative process helps filter noise and focus on relevant semantic anchors.

The framework is optimized using a Contrastive Loss (NT-Xent), which maximizes the similarity between paired audio and text embeddings while minimizing similarity for mismatched pairs.

3. Experimental Results

The model was evaluated on the Clotho (approx. 5,000 samples) and AudioCaps (approx. 50,000 samples) datasets.

3.1 Performance Metrics

The co-attention models demonstrated substantial improvements over the previous state-of-the-art (SOTA) baselines:

  • Clotho Dataset: The Iterating model achieved an mAP@10 of 30.9, a 16.6% improvement over the GPTtar baseline (26.5).
  • AudioCaps Dataset: The model reached an mAP@10 of 37.4, a 15.1% improvement over the baseline (32.5).
  • Recall: On AudioCaps, the R@1 (Recall at rank 1) increased from 36.1% to 41.5%, indicating a significantly higher precision in identifying the exact target audio.

3.2 Key Insights

The experiments yielded several critical findings:

  1. Bidirectionality: Simultaneous attention (audio-to-text and text-to-audio) is superior to unidirectional approaches.
  2. Iterative Refinement: The Iterating Module consistently outperformed the Stacking Module. The authors suggest that hierarchical refinement is more effective at handling the initial noise present in raw embeddings.
  3. Depth of Reasoning: Performance scaled positively with network depth (up to 5 layers), suggesting that complex audio-text alignment requires deep, multi-stage processing.

4. Conclusion and Future Work

This research demonstrates that cascaded co-attention networks significantly enhance the accuracy of language-based audio retrieval. By facilitating deep, bidirectional interactions between text and sound, the model effectively bridges the heterogeneity gap.

While effective, the authors note that the increased computational complexity of deep cascaded layers remains a challenge. Future work will focus on improving the scalability of these architectures for "in-the-wild" datasets containing millions of audio clips, where noise and alignment sensitivity become even more critical factors.

Summary updated

Abstract

In recent years, user-generated audio content has proliferated across various media platforms, creating a growing need for efficient retrieval methods that allow users to search for audio clips using natural language queries. This task, known as language-based audio retrieval, presents significant challenges due to the complexity of learning semantic representations from heterogeneous data across both text and audio modalities. In this work, we introduce a novel framework for the language-based audio retrieval task that leverages co-attention mechanismto jointly learn meaningful representations from both modalities. To enhance the model’s ability to capture fine-grained cross-modal interactions, we propose a cascaded co-attention architecture, where co-attention modules are stacked or iterated to progressively refine the semantic alignment between text and audio. Experiments conducted on two public datasets show that the proposed method can achieve better performance than the state-of-the-art method. Specifically, our best performed co-attention model achieves a 16.6% improvement in mean Average Precision on Clotho dataset, and a $\mathbf{1 5. 1 \%}$ improvement on AudioCaps.
BibTeX
@misc{sun2024language,
  author = {Haoran Sun and Zimu Wang and Qiuyi Chen and Jianjun Chen and Jia Wang and Haiyang Zhang},
  title = {{Language-based Audio Retrieval with Co-Attention Networks}},
  year = {2024},
  eprint = {2412.20914},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.20914},
}
Figure