CST-former: Multidimensional Attention-based Transformer for Sound Event Localization and Detection in Real Scenes

Yusun Shul, Dayun Choi, Jung‐Woo ChoiarXiv 2025 · DOI · arXiv

TL;DR

Researchers from KAIST have developed CST-former, a Transformer-based model that achieves state-of-the-art performance in Sound Event Localization and Detection (SELD) within complex, real-world environments. By applying specialized attention mechanisms across the Channel, Spectro, and Temporal domains, and introducing a multiscale embedding technique, the model accurately identifies "what" a sound is and "where" it's coming from without requiring massive external datasets.

What Problem Does This Paper Address?

Sound Event Localization and Detection (SELD) is the "eyes and ears" of machine hearing. It requires a system to simultaneously classify a sound (e.g., a dog barking) and estimate its Direction of Arrival (DoA) in 3D space.

While deep learning has made strides here, real-world scenes present three major headaches:

  1. Domain Complexity: Audio data contains vital information hidden across time, frequency, and spatial (channel) dimensions. Most existing models focus heavily on the temporal dimension, often neglecting the others.
  2. Data Scarcity: Real-world annotated SELD data (like the STARSS datasets) is difficult to collect. Many top-performing models rely on massive amounts of external data to generalize.
  3. Inconsistency: Models often struggle with "track permutation" (confusing which sound is which over time) and "false negatives" during inference in noisy environments.

Methods

The authors propose the CST-former, which moves beyond simple temporal attention.

1. The CST Block

The core of the architecture is a "sandwich" of three components:

  • Local Perception Unit (LPU): Uses depth-wise convolutions to capture immediate local features.
  • CST Attention: This is the heart of the paper. It applies three independent attention heads:
  • Channel (C): Learns spatial relationships between microphone signals.
  • Spectral (S): Focuses on frequency-based characteristics.
  • Temporal (T): Captures how the sound evolves over time.
  • Inverted Residual Feed-Forward Network (IRFFN): Shuffles and aggregates these features to prepare them for the next layer.

2. MSULE (Multiscale Unfolded Local Embedding)

To feed the Channel attention, the authors developed ULE, which "unfolds" local time-frequency patches into embeddings. MSULE takes this further by using varying kernel sizes across different layers, allowing the model to capture both fine-grained details and broader global contexts of a sound.

3. VTM and CTAI (The "Secret Sauce")

  • Vector Threshold Masking (VTM): A finetuning trick. During training, it masks outputs that fall below the detection threshold, forcing the model to "shout louder" (increase vector length) for real events, which reduces missed detections (False Negatives).
  • Clustered-track Augmented Inference (CTAI): A post-processing step that uses K-means clustering to resolve inconsistencies when the model looks at the same audio from different rotated perspectives (Test-Time Augmentation).

Datasets

The model was evaluated on the primary benchmarks for real-world SELD:

  • STARSS22 & STARSS23: These contain real-world recordings from various rooms with human-made sounds.
  • Synth-set: A synthetic dataset used for pre-training to supplement the limited real-world recordings.

Key Insights and Ideas

  • Spatial Info is in the Channels: By visualizing attention maps, the authors proved that Channel attention specifically captures localization-related features. When they perturbed the source location, the Channel attention maps shifted predictably.
  • Order Matters: Through ablation studies, they found that the sequence C → S → T was optimal. Placing Channel attention first allows the model to organize spatial embeddings before refining them through frequency and time.
  • Multiscale is Mandatory: Sounds like "clapping" (short/impulsive) and "speech" (long/harmonic) occupy different scales in a spectrogram. MSULE allows the model to adapt its "zoom level" to catch both.

Key Results

CST-former achieved remarkable results without using any external data:

  • STARSS22: The "Large" version of CST-former achieved an $S_{SELD}$ score of 0.3066, a 26% improvement over the DCASE 2023 baseline.
  • SOTA Comparison: It outperformed much larger models, such as EINV2 (85M parameters) and ResNet-Conformer (13M parameters), while the CST-former (Base) uses only 0.39M parameters.
  • Robustness: The model showed significant performance gains in detecting impulsive sounds (like telephones or knocks) which are traditionally difficult for temporal-only models.

Limitations and Future Work

  • Room Dependency: While the model generalized well to most unseen rooms, performance in "Room 2" of the STARSS dataset remained lower than others, suggesting that extremely high reverberation or specific acoustic profiles still pose a challenge.
  • Model Scaling: The "Huge" version of the model actually showed slight performance degradation compared to the "Large" version, indicating a point of diminishing returns where the model begins to overfit the limited training data.
  • Real-time Constraints: While the paper focuses on accuracy, the computational overhead of multidimensional attention in a real-time edge device (like a smart speaker) remains an area for further optimization.

Summary updated

Abstract

Sound event localization and detection (SELD) is a task for the classification of sound events and the identification of direction of arrival (DoA) utilizing multichannel acoustic signals. For effective classification and localization, a channel-spectro-temporal transformer (CST-former) was suggested. CST-former employs multidimensional attention mechanisms across the spatial, spectral, and temporal domains to enlarge the model's capacity to learn the domain information essential for event detection and DoA estimation over time. In this work, we present an enhanced version of CST-former with multiscale unfolded local embedding (MSULE) developed to capture and aggregate domain information over multiple time-frequency scales. Also, we propose finetuning and post-processing techniques beneficial for conducting the SELD task over limited training datasets. In-depth ablation studies of the proposed architecture and detailed analysis on the proposed modules are carried out to validate the efficacy of multidimensional attentions on the SELD task. Empirical validation through experimentation on STARSS22 and STARSS23 datasets demonstrates the remarkable performance of CST-former and post-processing techniques without using external data.
BibTeX
@misc{shul2025cst,
  author = {Yusun Shul and Dayun Choi and Jung‐Woo Choi},
  title = {{CST-former: Multidimensional Attention-based Transformer for Sound Event Localization and Detection in Real Scenes}},
  year = {2025},
  eprint = {2504.12870},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2504.12870},
  howpublished = {arXiv},
}
Figure