SAS: Scene-Annotated AudioSet Strong for Joint Scene-Event Analysis

Michel Olvera, Paraskevas Stamatiadis, Romain Serizel, Slim Essid, Gaël RichardarXiv 2026 · arXiv

TL;DR

The authors introduce SAS (Scene-Annotated AudioSet Strong), a large-scale dataset that adds clip-level scene annotations to over 200,000 videos from AudioSet Strong. By leveraging a cross-modal labeling pipeline and a corpus-scale analysis, they demonstrate that an acoustic scene's identity is better defined by its distinctive sounds rather than its most frequent ones, paving the way for more robust context-aware machine listening.

What Problem Does This Paper Address?

In the real world, we never hear sounds in a vacuum. A "hissing" sound is a non-event in a kitchen (a kettle) but a cause for alarm in a garage (a gas leak). Human auditory perception relies heavily on this environmental context to resolve ambiguities.

However, machine listening research has been hampered by a data gap. We have massive datasets for Sound Event Detection (SED) like AudioSet, and separate datasets for Acoustic Scene Classification (ASC). What’s missing is a large-scale resource that links the two. Without it, models struggle to learn the statistical relationships between events and their surroundings, making it difficult to build systems that "understand" context the way humans do.

Methods

To bridge this gap, the researchers developed a four-stage cross-modal labeling pipeline to annotate the existing AudioSet Strong dataset:

  1. Visual Proxy Extraction: They extracted both uniform (1 Hz) and content-aware (I-frame) frames from the videos.
  2. Candidate Generation: Three vision models (ResNet50, WaveMix, and LLaVA) processed these frames to suggest potential scene labels based on the Places365 taxonomy.
  3. LLM Reasoning: This is the "brain" of the pipeline. Five different Large Language Models (including Mistral and Llama) were given a prompt containing the visual candidates plus the ground-truth sound event labels already present in AudioSet Strong. The LLMs reasoned across these modalities to suggest a final scene.
  4. Ensemble Voting: A majority vote across the LLMs determined the final label, which was then mapped to two new taxonomies: Audiovisual (AV) for general scenes and Audio-centric (AC) for scenes that are visually distinct but acoustically similar (e.g., grouping "marsh" and "swamp" into "wetland").
Fig. 2: Overview of the acoustic scene label extraction pipeline: visual evidence extraction, cross-modal reasoning, ensemble voting, and taxonomy mapping.

Datasets

  • AudioSet Strong: The foundational dataset providing temporally localized sound event labels.
  • SAS (Scene-Annotated AudioSet Strong): The new benchmark introduced in this paper, containing scene annotations for 207,315 unique video identifiers.
  • Places365: Used for pre-training the vision models' scene recognition capabilities.

Key Insights and Ideas

  • Distinctiveness > Frequency: A major "aha" moment in the paper is that raw event frequency is a poor predictor of scene identity. Common sounds like "speech" or "footsteps" happen everywhere. Instead, scene identity is driven by discriminative events—sounds that are unique to that environment (e.g., a "dental drill" in a dental office).
  • Acoustic Chameleons vs. Semantic Anchors: The authors identify "acoustic chameleons"—scenes like markets or skate parks whose raw acoustic profiles are dominated by generic background noise, making them hard to identify by sound alone. In contrast, "semantic anchors" like offices or studios have acoustic profiles that align closely with their semantic definitions.
  • Geometric Divergence: By comparing "acoustic space" (how similar scenes sound) to "semantic space" (how similar their labels are), the authors found that divergence between these two predicts where machine learning models are likely to fail.
Fig. 3: 2-D projections of representational views with highlighted neighborhoods of four scenes across acoustic, semantic, and lexical spaces.

Key Results

  • Scale: SAS provides over 200k annotations, including 130,000 single-label and 12,000 multi-label clips.
  • Quality: Manual validation showed a Cohen’s Kappa of 0.634, indicating substantial agreement between the automated pipeline and human annotators.
  • Classification Performance: Using a Naive Bayes classifier, the authors showed that weighting events by their "semantic importance" (using dPPMI) improved Macro-F1 scores (from 0.1282 to 0.1315 on the AV taxonomy), particularly for rare or easily confused scenes.
  • Alignment: Semantic profiles (weighted by distinctiveness) aligned significantly better with human-derived taxonomies than raw acoustic profiles (0.312 vs 0.123 correlation).

Limitations and Future Work

The authors acknowledge that while the pipeline is robust, it still relies on visual proxies. If a video's visual content is misleading or disconnected from its audio, the scene label might be inaccurate. Additionally, the dataset follows a long-tail distribution, with some scenes having far more examples than others, which remains a challenge for training balanced models.

Future work suggests using SAS to develop adaptive classifiers that can decide on-the-fly whether to trust raw acoustic evidence or rely more heavily on semantic context based on the "neighborhood instability" of the detected scene.

Summary updated

Abstract

Context is central to human auditory perception, as listeners interpret sounds within their acoustic environment to resolve ambiguity. However, large-scale datasets that jointly capture sound events and acoustic scenes remain limited. We address this by augmenting AudioSet Strong with clip-level scene annotations under two taxonomies for joint sceneevent modeling. A corpus-scale analysis shows that scene identity is better captured by events distinctive to a scene than by those that merely occur frequently. Reweighting event co-occurrences toward scene-specific associations produces representations that better align with human-derived taxonomies, while also exposing scenes where the acoustic and semantic views diverge. This geometric divergence predicts classification errors, while supervised validation confirms that the proposed scene annotations provide meaningful complementary information for developing scalable context-aware machine listening systems.
BibTeX
@misc{olvera2026sas,
  author = {Michel Olvera and Paraskevas Stamatiadis and Romain Serizel and Slim Essid and Gaël Richard},
  title = {{SAS: Scene-Annotated AudioSet Strong for Joint Scene-Event Analysis}},
  year = {2026},
  eprint = {2611.09199},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09199},
}
Figure