Learning Audio Concepts from Counterfactual Natural Language

Ali Vosoughi, Luca Bondi, Ho-Hsiang Wu, Chenliang Xu

Introduction

State-of-the-art audio-text representation models, such as Contrastive Language-Audio Pretraining (CLAP), have demonstrated significant capabilities in matching audio clips with natural language descriptions. However, these models frequently struggle with fine-grained discrimination. For instance, a model might fail to distinguish between the sound of a gunshot and a firework when both occur within the same ambient context of a crowded outdoor event.

The primary bottleneck in resolving this issue is data scarcity. It is practically and ethically challenging to record "controlled trials" of audio where only a single variable is altered while all other environmental factors remain constant. Consequently, existing datasets primarily describe what is present in an audio clip, lacking the "negative" examples necessary to teach a model what is absent. This paper introduces a novel framework that utilizes Large Language Models (LLMs) to generate counterfactual natural language as a proxy for these missing experimental audio trials.

Methodology

The authors propose a two-step pipeline designed to infuse causal reasoning into audio-text representation learning.

1. Counterfactual Generation via LLMs

Using GPT-3.5-Turbo and Chain-of-Thought (CoT) prompting, the system analyzes human-written captions to identify specific sound sources, categorized as Primary, Background, or Ambient. The LLM then performs a linguistic "intervention," altering one of these sources to create a counterfactual caption. These generated captions are designed to be linguistically plausible but factually incorrect for the specific audio clip (e.g., swapping a "barking dog" for a "revving car" while maintaining the same background description).

2. Causal Learning Framework

The model employs a triplet-style loss architecture to refine the embedding space, consisting of two primary components:

  • Factual Consistency Loss: This ensures that the audio embedding remains close to the original, human-annotated factual caption.
  • Angle Loss: This novel loss term maximizes the angular distance between the audio embedding and the generated counterfactual caption.

By pushing the audio embedding away from these "hard negative" counterfactuals, the model is forced to learn the specific acoustic features that validate the factual caption, effectively "sharpening" its perception of audio concepts.

Experimental Setup

The framework was trained and evaluated using several major human-annotated datasets:

  • Training/Evaluation: AudioCaps (~44k pairs), Clotho (~30k pairs), and MACS (~17k pairs).
  • Downstream Tasks: Zero-shot classification performance was tested on the ESC-50 (Environmental Sound Classification) and UrbanSound8K (US8K) datasets.

Key Insights

LLMs as Causal Proxies

The research demonstrates that LLMs can serve as effective knowledge bases for imagining alternative scenarios. This approach circumvents the physical impossibility of collecting perfectly controlled counterfactual audio data, allowing the model to learn from "what-if" scenarios that do not exist in the training set.

The Nuance Trade-off

The study identifies a trade-off between factual consistency and discriminability. While standard models prioritize proximity to the truth, the introduction of counterfactuals forces the model to understand why a description is true. t-SNE visualizations of the embedding space indicate that audio embeddings move significantly further from false captions and align more tightly with true ones as counterfactual loss is introduced.

Results

  • Audio Retrieval: On the Clotho evaluation set, the proposed method achieved a top-1 accuracy of 0.126, a relative improvement of over 43% compared to the standard CLAP baseline (0.088).
  • Zero-Shot Classification: The model outperformed existing benchmarks (Wav2CLIP, AudioClip, and CLAP) on the ESC-50 dataset, reaching an accuracy of 0.744.
  • Ablation Studies: Results confirmed that the combination of factual consistency and angle loss provides the optimal balance, preventing model confusion while maximizing discriminative power.

Limitations and Future Work

Despite its successes, the model showed a performance decline on the UrbanSound8K dataset (0.475 vs. CLAP's 0.798). The authors suggest this is due to US8K's limited class count and lack of textual detail; the model's learned nuances may be counterproductive when labels are overly simplistic or highly similar (e.g., "siren" vs. generic street sounds). Additionally, the frozen CLIP text encoder remains a bottleneck, as it often fails to capture fine-grained semantic nuances (e.g., high BERTscore similarity between "dog" and "cat").

Future research will focus on exploring more complex levels of counterfactual reasoning and integrating more sophisticated text encoders to further refine the distinction between similar acoustic concepts.

Summary updated

BibTeX
@inproceedings{vosoughi2024learning,
  author = {Ali Vosoughi and Luca Bondi and Ho-Hsiang Wu and Chenliang Xu},
  title = {{Learning Audio Concepts from Counterfactual Natural Language}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10446736},
}
Figure