TL;DR
The authors introduce SAS (Scene-Annotated AudioSet Strong), a large-scale dataset that adds clip-level scene annotations to over 200,000 videos from AudioSet Strong. By leveraging a cross-modal labeling pipeline and a corpus-scale analysis, they demonstrate that an acoustic scene's identity is better defined by its distinctive sounds rather than its most frequent ones, paving the way for more robust context-aware machine listening.
What Problem Does This Paper Address?
In the real world, we never hear sounds in a vacuum. A "hissing" sound is a non-event in a kitchen (a kettle) but a cause for alarm in a garage (a gas leak). Human auditory perception relies heavily on this environmental context to resolve ambiguities.
However, machine listening research has been hampered by a data gap. We have massive datasets for Sound Event Detection (SED) like AudioSet, and separate datasets for Acoustic Scene Classification (ASC). What’s missing is a large-scale resource that links the two. Without it, models struggle to learn the statistical relationships between events and their surroundings, making it difficult to build systems that "understand" context the way humans do.
Methods
To bridge this gap, the researchers developed a four-stage cross-modal labeling pipeline to annotate the existing AudioSet Strong dataset:
- Visual Proxy Extraction: They extracted both uniform (1 Hz) and content-aware (I-frame) frames from the videos.
- Candidate Generation: Three vision models (ResNet50, WaveMix, and LLaVA) processed these frames to suggest potential scene labels based on the Places365 taxonomy.
- LLM Reasoning: This is the "brain" of the pipeline. Five different Large Language Models (including Mistral and Llama) were given a prompt containing the visual candidates plus the ground-truth sound event labels already present in AudioSet Strong. The LLMs reasoned across these modalities to suggest a final scene.
- Ensemble Voting: A majority vote across the LLMs determined the final label, which was then mapped to two new taxonomies: Audiovisual (AV) for general scenes and Audio-centric (AC) for scenes that are visually distinct but acoustically similar (e.g., grouping "marsh" and "swamp" into "wetland").
Datasets
- AudioSet Strong: The foundational dataset providing temporally localized sound event labels.
- SAS (Scene-Annotated AudioSet Strong): The new benchmark introduced in this paper, containing scene annotations for 207,315 unique video identifiers.
- Places365: Used for pre-training the vision models' scene recognition capabilities.
Key Insights and Ideas
- Distinctiveness > Frequency: A major "aha" moment in the paper is that raw event frequency is a poor predictor of scene identity. Common sounds like "speech" or "footsteps" happen everywhere. Instead, scene identity is driven by discriminative events—sounds that are unique to that environment (e.g., a "dental drill" in a dental office).
- Acoustic Chameleons vs. Semantic Anchors: The authors identify "acoustic chameleons"—scenes like markets or skate parks whose raw acoustic profiles are dominated by generic background noise, making them hard to identify by sound alone. In contrast, "semantic anchors" like offices or studios have acoustic profiles that align closely with their semantic definitions.
- Geometric Divergence: By comparing "acoustic space" (how similar scenes sound) to "semantic space" (how similar their labels are), the authors found that divergence between these two predicts where machine learning models are likely to fail.
Key Results
- Scale: SAS provides over 200k annotations, including 130,000 single-label and 12,000 multi-label clips.
- Quality: Manual validation showed a Cohen’s Kappa of 0.634, indicating substantial agreement between the automated pipeline and human annotators.
- Classification Performance: Using a Naive Bayes classifier, the authors showed that weighting events by their "semantic importance" (using dPPMI) improved Macro-F1 scores (from 0.1282 to 0.1315 on the AV taxonomy), particularly for rare or easily confused scenes.
- Alignment: Semantic profiles (weighted by distinctiveness) aligned significantly better with human-derived taxonomies than raw acoustic profiles (0.312 vs 0.123 correlation).
Limitations and Future Work
The authors acknowledge that while the pipeline is robust, it still relies on visual proxies. If a video's visual content is misleading or disconnected from its audio, the scene label might be inaccurate. Additionally, the dataset follows a long-tail distribution, with some scenes having far more examples than others, which remains a challenge for training balanced models.
Future work suggests using SAS to develop adaptive classifiers that can decide on-the-fly whether to trust raw acoustic evidence or rely more heavily on semantic context based on the "neighborhood instability" of the detected scene.
Summary updated