Unified Audio Event Detection

Yidi Jiang, Ruijie Tao, Wen Huang, Qian Chen, Wen Wang

TL;DR

This paper introduces Unified Audio Event Detection (UAED), a novel task that simultaneously identifies non-speech sound events and tracks fine-grained speech segments based on individual speaker identities. To solve this, the authors propose T-UAED, a Transformer-based framework that leverages specialized auditory encoders and a query-based decoder to capture the synergistic interactions between sound detection and speaker diarization. The unified model significantly outperforms naive pipeline combinations and achieves performance comparable to state-of-the-art specialized models on standard benchmarks.

What Problem Does This Paper Address?

In audio analysis, researchers typically split the world into two distinct tasks:

  1. Sound Event Detection (SED): Identifying what sound is happening and when (e.g., a dog barking or a vacuum cleaner running). In SED, all human speech is lumped into a single generic "speech" category.
  2. Speaker Diarization (SD): Answering the question of who spoke when in a multi-speaker environment. In SD, any non-speech sound is dismissed as irrelevant background noise.

However, real-world acoustic environments—like a bustling factory floor, a noisy train, or a lively dinner party—are complex mixtures of both. Treating speech as a monolith or ignoring background sounds provides an incomplete picture.

The authors argue that SED and SD are fundamentally synergistic. Knowing that a machine is running in the background can help a model better isolate and track speaker boundaries; conversely, precisely tracking individual speakers improves overall speech detection. This paper bridges the gap by defining the UAED task to provide a holistic, frame-level understanding of the entire soundscape.

Fig. 1: Illustration of the differences between our proposed UAED task and the traditional SED and SD tasks. UAED aims to simultane- ously identify the regions of both non-speech sound events and speech segments from individual speakers, thereby providing a holistic view of the soundscape.

Methods

The authors propose T-UAED (Transformer-based Unified Audio Event Detection), an encoder-decoder architecture designed to handle both tasks in parallel.

Fig. 2: The overview of our Transformer-based unified Audio Event Detection (T-UAED) framework. It outputs the occurrence regions for each audio event comprising both non-speech sound events and speaker-aware speech events. In this framework, ⊕symbolizes feature fusion layer. ⊗and σ represent the dot product and sigmoid operation, respectively.

1. Front-End Auditory Encoders

To capture both general acoustic properties and speaker-specific traits, T-UAED combines three distinct encoders:

  • CNN Sound Module: Captures local spectral and temporal variations.
  • BEATs Encoder: A self-supervised pretrained model that excels at capturing high-level semantic sound information.
  • WavLM Encoder: A self-supervised pretrained model optimized for robust speech and speaker representations.

To keep the model computationally efficient, the pretrained BEATs and WavLM encoders are frozen, and lightweight sound and speaker adapters (consisting of 1D convolutional layers and bottleneck structures) are introduced for parameter-efficient fine-tuning.

2. UAED Queries & Transformer Decoder

Instead of using separate classification heads, the model uses a query-based approach inspired by modern object detection networks (like DETR).

  • Sound Queries: Learnable embedding vectors assigned to each predefined non-speech sound category (e.g., cat, water).
  • Speaker Queries: High-quality speaker embeddings extracted from pre-enrolled speech samples using a pretrained ECAPA-TDNN encoder.

The Transformer decoder uses cross-attention to match these queries against the frame-level audio features generated by the encoder. Finally, a dot product between the encoder and decoder outputs, followed by a sigmoid operation, yields frame-by-frame binary probabilities for every audio event.

Datasets

Because no real-world dataset contains comprehensive labels for both fine-grained speaker identities and diverse sound events, the authors developed an online data simulation strategy to construct UAED Data:

  • Speech Component: 3-speaker conversations simulated from the Librispeech dataset, using conversational statistics from the DIHARD II development set (500 hours for training, 5 hours for evaluation).
  • Sound Component: Background and foreground sound events randomly sampled from 9 classes in the DESED soundbank (e.g., alarm ringing, frying, blender) and mixed into the speech tracks at varying signal-to-noise ratios (SNRs).

For downstream validation on individual tasks, the authors evaluated the model on the DESED dataset (for SED) and the CALLHOME dataset (for SD).

Key Insights and Ideas

  • Noise is Information: The core "aha moment" is that background noise shouldn't be ignored during speaker diarization. Explicitly modeling non-speech events prevents the diarization system from confusing background noises with speech activity, leading to cleaner speaker boundaries.
  • Parallel Cross-Attention Overcomes Overlap: In complex audio, sounds and speech frequently overlap. By using parallel queries in a Transformer decoder, the model naturally handles polyphonic audio events without needing complex, hand-crafted overlapping speech heuristics.

Key Results

The authors evaluated T-UAED against single-task baselines and a pipeline baseline (Model A+B) that simply concatenates separate SED and SD model outputs.

Joint UAED Performance

T-UAED significantly outperformed the pipeline baseline on the joint task, proving the benefit of joint modeling:

  • Intersection-Based F1 (IB-F1): 0.652 (T-UAED) vs. 0.608 (Baseline)
  • Segment-Based F1 (SB-F1): 0.824 (T-UAED) vs. 0.796 (Baseline)

Speaker Diarization (SD) Versatility

When evaluated on the standard CALLHOME dataset, the speaker branch of the model (T-UAED-SD) achieved state-of-the-art results compared to competitive specialized diarization systems.

TABLE III: DER (%) results comparison on the CALLHOME Part 2 2-spk and 3-spk subsets. Lower is better. T-UAED-SD (oracle) de- notes T-UAED-SD with the ground-truth enrolled speech as speaker embedding for speaker diarization, T-UAED-SD (cluster) denotes T-UAED-SD with speaker embeddings extracted from the estimated reference speech by clustering front-end.

Using ground-truth enrolled speech (oracle), T-UAED achieved a Diarization Error Rate (DER) of 6.80% on 2-speaker data and 8.51% on 3-speaker data, outperforming well-known architectures like SA-EEND and TS-VAD.

Sound Event Detection (SED) Versatility

On the DESED benchmark, the sound branch (T-UAED-SED) achieved a Polyphonic Sound Detection Score (PSDS1) of 0.517, delivering comparable performance to specialized state-of-the-art SED models without requiring extra encoder fine-tuning on massive external datasets.

Limitations and Future Work

The primary limitation acknowledged by the authors is that the current framework operates under a closed-set assumption. It requires knowing the specific categories of non-speech sounds in advance and relies on a fixed number of pre-enrolled speakers.

Future research will focus on expanding the framework's generalization capabilities to handle open-set scenarios—enabling the system to dynamically detect unseen sound classes and cluster an unknown number of novel speakers on the fly.

Summary updated

Abstract

Sound Event Detection (SED) detects regions of sound events, while Speaker Diarization (SD) segments speech conversations attributed to individual speakers. In SED, all speaker segments are classified as a single speech event, while in SD, non-speech sounds are treated merely as background noise. Thus, both tasks provide only partial analysis in complex audio scenarios involving both speech conversation and non-speech sounds. In this paper, we introduce a novel task called Unified Audio Event Detection (UAED) for comprehensive audio analysis. UAED explores the synergy between SED and SD tasks, simultaneously detecting non-speech sound events and fine-grained speech events based on speaker identities. To tackle this task, we propose a Transformer-based UAED (T-UAED) framework and construct the UAED Data derived from the Librispeech dataset and DESED soundbank. Experiments demonstrate that the proposed framework effectively exploits task interactions and substantially outperforms the baseline that simply combines the outputs of SED and SD models. T-UAED also shows its versatility by performing comparably to specialized models for individual SED and SD tasks on DESED and CALLHOME datasets.
BibTeX
@inproceedings{jiang2025unified,
  author = {Yidi Jiang and Ruijie Tao and Wen Huang and Qian Chen and Wen Wang},
  title = {{Unified Audio Event Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888080},
}
Figure