Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning

Manh Luong, Khai Nguyen, Dinh Phung, Gholamreza Haffari, Lizhen Qu

Introduction

Audio captioning—the task of describing a sound clip in natural language—is a critical component of modern audio-to-text systems. However, these models often suffer from exposure bias. During training, models typically use "teacher forcing," where they are given ground-truth words at each step. During inference, they must rely on their own previously generated outputs, which can lead to generic, repetitive, or inaccurate descriptions.

Furthermore, existing contrastive learning techniques often rely on global feature averages, ignoring the temporal relationship between sounds. To address these challenges, researchers introduced the ACUS (Audio Captioning with Unbiased sliced Wasserstein kernel) framework at NeurIPS 2025.

The ACUS Framework

The ACUS framework utilizes a mathematically unbiased "Sliced Wasserstein" kernel to align audio and text features while preserving chronological accuracy.

Overview of the ACUS framework during training and inference phases.

Technical Innovations

  1. The USW-RBF Kernel: Standard Sliced Wasserstein (SW) kernels used in Radial Basis Functions (RBF) are often biased because the mathematical expectation is placed inside an exponential function. ACUS introduces the Unbiased Sliced Wasserstein RBF (USW-RBF), where the expectation is moved outside. This ensures the kernel is positive definite and reduces approximation error at a reliable parametric rate ($O(L^{-1/2})$), making it highly compatible with stochastic gradient optimization.
  2. Temporal Awareness: To ensure the model respects the timing of sounds, the kernel is equipped with Rotary Positional Embeddings (RoPE). This allows the Wasserstein distance—a metric measuring the cost of transforming one distribution into another—to account for the specific sequence and timing of acoustic events.
  3. Stochastic Reranking: Instead of simple greedy search, the framework uses stochastic decoding (such as nucleus sampling) to generate multiple candidate captions. The USW-RBF score then reranks these candidates to select the one that best aligns with the audio's temporal and acoustic features.

Experimental Setup

The model was evaluated on two gold-standard datasets:

  • AudioCaps: A large-scale dataset of YouTube clips with human-written captions.
  • Clotho: A diverse dataset focused on everyday sounds with high lexical variety.

To test generalizability, the kernel was also evaluated on audio reasoning benchmarks, including CompA-R (scored by GPT-4o) and the MMAU multi-task audio understanding benchmark.

Key Results

The ACUS framework outperformed state-of-the-art baselines across nearly all metrics.

Quantitative performance comparison of ACUS against state-of-the-art methods.

On the AudioCaps dataset, the model achieved a SPIDEr score of 0.5, a significant improvement over the Enclap baseline (0.48). Retrieval accuracy also saw notable gains; text-to-audio retrieval (R@1) improved from 29.2 to 32.2, demonstrating that the generated captions are more representative of the actual audio content. Qualitatively, captions became more descriptive, capturing environmental context that baseline models often missed.

Limitations and Future Work

The primary trade-off for increased caption quality is computational overhead. Because the system generates and reranks approximately 30 candidate captions, the inference time is longer than standard greedy search. The Real-Time Factor (RTF) increased from 0.33 to 0.81. Future work will explore how this kernel scales with multi-billion parameter audio-language models and methods to optimize inference latency.

The authors have released their code and framework on GitHub.

Summary updated

Abstract

Audio captioning systems face a fundamental challenge: teacher-forcing training creates exposure bias that leads to caption degeneration during inference. While contrastive methods have been proposed as solutions, they typically fail to capture the crucial temporal relationships between acoustic and linguistic modalities. We address this limitation by introducing the unbiased sliced Wasserstein RBF (USW-RBF) kernel with rotary positional embedding, specifically designed to preserve temporal information across modalities. Our approach offers a practical advantage: the kernel enables efficient stochastic gradient optimization, making it computationally feasible for real-world applications. Building on this foundation, we develop a complete audio captioning framework that integrates stochastic decoding to further mitigate caption degeneration. Extensive experiments on AudioCaps and Clotho datasets demonstrate that our method significantly improves caption quality, lexical diversity, and text-to-audio retrieval accuracy. Furthermore, we demonstrate the generalizability of our USW-RBF kernel by applying it to audio reasoning tasks, where it enhances the reasoning capabilities of large audio language models on the CompA-R in terms of correctness and quality. Our kernel also improves the reasoning accuracy of the MMAU-test-mini benchmarks by $4\%$. These results establish our approach as a powerful and generalizable solution for cross-modal alignment challenges in audio-language tasks.
BibTeX
@inproceedings{luong2025unbiased,
  author = {Manh Luong and Khai Nguyen and Dinh Phung and Gholamreza Haffari and Lizhen Qu},
  title = {{Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=83HlJrZigZ},
}
Figure