Introduction
Audio captioning—the task of describing a sound clip in natural language—is a critical component of modern audio-to-text systems. However, these models often suffer from exposure bias. During training, models typically use "teacher forcing," where they are given ground-truth words at each step. During inference, they must rely on their own previously generated outputs, which can lead to generic, repetitive, or inaccurate descriptions.
Furthermore, existing contrastive learning techniques often rely on global feature averages, ignoring the temporal relationship between sounds. To address these challenges, researchers introduced the ACUS (Audio Captioning with Unbiased sliced Wasserstein kernel) framework at NeurIPS 2025.
The ACUS Framework
The ACUS framework utilizes a mathematically unbiased "Sliced Wasserstein" kernel to align audio and text features while preserving chronological accuracy.
Technical Innovations
- The USW-RBF Kernel: Standard Sliced Wasserstein (SW) kernels used in Radial Basis Functions (RBF) are often biased because the mathematical expectation is placed inside an exponential function. ACUS introduces the Unbiased Sliced Wasserstein RBF (USW-RBF), where the expectation is moved outside. This ensures the kernel is positive definite and reduces approximation error at a reliable parametric rate ($O(L^{-1/2})$), making it highly compatible with stochastic gradient optimization.
- Temporal Awareness: To ensure the model respects the timing of sounds, the kernel is equipped with Rotary Positional Embeddings (RoPE). This allows the Wasserstein distance—a metric measuring the cost of transforming one distribution into another—to account for the specific sequence and timing of acoustic events.
- Stochastic Reranking: Instead of simple greedy search, the framework uses stochastic decoding (such as nucleus sampling) to generate multiple candidate captions. The USW-RBF score then reranks these candidates to select the one that best aligns with the audio's temporal and acoustic features.
Experimental Setup
The model was evaluated on two gold-standard datasets:
- AudioCaps: A large-scale dataset of YouTube clips with human-written captions.
- Clotho: A diverse dataset focused on everyday sounds with high lexical variety.
To test generalizability, the kernel was also evaluated on audio reasoning benchmarks, including CompA-R (scored by GPT-4o) and the MMAU multi-task audio understanding benchmark.
Key Results
The ACUS framework outperformed state-of-the-art baselines across nearly all metrics.
On the AudioCaps dataset, the model achieved a SPIDEr score of 0.5, a significant improvement over the Enclap baseline (0.48). Retrieval accuracy also saw notable gains; text-to-audio retrieval (R@1) improved from 29.2 to 32.2, demonstrating that the generated captions are more representative of the actual audio content. Qualitatively, captions became more descriptive, capturing environmental context that baseline models often missed.
Limitations and Future Work
The primary trade-off for increased caption quality is computational overhead. Because the system generates and reranks approximately 30 candidate captions, the inference time is longer than standard greedy search. The Real-Time Factor (RTF) increased from 0.33 to 0.81. Future work will explore how this kernel scales with multi-billion parameter audio-language models and methods to optimize inference latency.
The authors have released their code and framework on GitHub.
Summary updated