TL;DR
This paper introduces Unified Audio Event Detection (UAED), a novel task that simultaneously identifies non-speech sound events and tracks fine-grained speech segments based on individual speaker identities. To solve this, the authors propose T-UAED, a Transformer-based framework that leverages specialized auditory encoders and a query-based decoder to capture the synergistic interactions between sound detection and speaker diarization. The unified model significantly outperforms naive pipeline combinations and achieves performance comparable to state-of-the-art specialized models on standard benchmarks.
What Problem Does This Paper Address?
In audio analysis, researchers typically split the world into two distinct tasks:
- Sound Event Detection (SED): Identifying what sound is happening and when (e.g., a dog barking or a vacuum cleaner running). In SED, all human speech is lumped into a single generic "speech" category.
- Speaker Diarization (SD): Answering the question of who spoke when in a multi-speaker environment. In SD, any non-speech sound is dismissed as irrelevant background noise.
However, real-world acoustic environments—like a bustling factory floor, a noisy train, or a lively dinner party—are complex mixtures of both. Treating speech as a monolith or ignoring background sounds provides an incomplete picture.
The authors argue that SED and SD are fundamentally synergistic. Knowing that a machine is running in the background can help a model better isolate and track speaker boundaries; conversely, precisely tracking individual speakers improves overall speech detection. This paper bridges the gap by defining the UAED task to provide a holistic, frame-level understanding of the entire soundscape.
Methods
The authors propose T-UAED (Transformer-based Unified Audio Event Detection), an encoder-decoder architecture designed to handle both tasks in parallel.
1. Front-End Auditory Encoders
To capture both general acoustic properties and speaker-specific traits, T-UAED combines three distinct encoders:
- CNN Sound Module: Captures local spectral and temporal variations.
- BEATs Encoder: A self-supervised pretrained model that excels at capturing high-level semantic sound information.
- WavLM Encoder: A self-supervised pretrained model optimized for robust speech and speaker representations.
To keep the model computationally efficient, the pretrained BEATs and WavLM encoders are frozen, and lightweight sound and speaker adapters (consisting of 1D convolutional layers and bottleneck structures) are introduced for parameter-efficient fine-tuning.
2. UAED Queries & Transformer Decoder
Instead of using separate classification heads, the model uses a query-based approach inspired by modern object detection networks (like DETR).
- Sound Queries: Learnable embedding vectors assigned to each predefined non-speech sound category (e.g., cat, water).
- Speaker Queries: High-quality speaker embeddings extracted from pre-enrolled speech samples using a pretrained ECAPA-TDNN encoder.
The Transformer decoder uses cross-attention to match these queries against the frame-level audio features generated by the encoder. Finally, a dot product between the encoder and decoder outputs, followed by a sigmoid operation, yields frame-by-frame binary probabilities for every audio event.
Datasets
Because no real-world dataset contains comprehensive labels for both fine-grained speaker identities and diverse sound events, the authors developed an online data simulation strategy to construct UAED Data:
- Speech Component: 3-speaker conversations simulated from the Librispeech dataset, using conversational statistics from the DIHARD II development set (500 hours for training, 5 hours for evaluation).
- Sound Component: Background and foreground sound events randomly sampled from 9 classes in the DESED soundbank (e.g., alarm ringing, frying, blender) and mixed into the speech tracks at varying signal-to-noise ratios (SNRs).
For downstream validation on individual tasks, the authors evaluated the model on the DESED dataset (for SED) and the CALLHOME dataset (for SD).
Key Insights and Ideas
- Noise is Information: The core "aha moment" is that background noise shouldn't be ignored during speaker diarization. Explicitly modeling non-speech events prevents the diarization system from confusing background noises with speech activity, leading to cleaner speaker boundaries.
- Parallel Cross-Attention Overcomes Overlap: In complex audio, sounds and speech frequently overlap. By using parallel queries in a Transformer decoder, the model naturally handles polyphonic audio events without needing complex, hand-crafted overlapping speech heuristics.
Key Results
The authors evaluated T-UAED against single-task baselines and a pipeline baseline (Model A+B) that simply concatenates separate SED and SD model outputs.
Joint UAED Performance
T-UAED significantly outperformed the pipeline baseline on the joint task, proving the benefit of joint modeling:
- Intersection-Based F1 (IB-F1): 0.652 (T-UAED) vs. 0.608 (Baseline)
- Segment-Based F1 (SB-F1): 0.824 (T-UAED) vs. 0.796 (Baseline)
Speaker Diarization (SD) Versatility
When evaluated on the standard CALLHOME dataset, the speaker branch of the model (T-UAED-SD) achieved state-of-the-art results compared to competitive specialized diarization systems.
Using ground-truth enrolled speech (oracle), T-UAED achieved a Diarization Error Rate (DER) of 6.80% on 2-speaker data and 8.51% on 3-speaker data, outperforming well-known architectures like SA-EEND and TS-VAD.
Sound Event Detection (SED) Versatility
On the DESED benchmark, the sound branch (T-UAED-SED) achieved a Polyphonic Sound Detection Score (PSDS1) of 0.517, delivering comparable performance to specialized state-of-the-art SED models without requiring extra encoder fine-tuning on massive external datasets.
Limitations and Future Work
The primary limitation acknowledged by the authors is that the current framework operates under a closed-set assumption. It requires knowing the specific categories of non-speech sounds in advance and relies on a fixed number of pre-enrolled speakers.
Future research will focus on expanding the framework's generalization capabilities to handle open-set scenarios—enabling the system to dynamically detect unseen sound classes and cluster an unknown number of novel speakers on the fly.
Summary updated