MTDA-HSED: Mutual-Assistance Tuning and Dual-Branch Aggregating for Heterogeneous Sound Event Detection

Zehao Wang, Haobo Yue, Zhicheng Zhang, Da Mu, Jin Tang, Jianqin Yin

Sound Event Detection (SED) is the task of not just identifying what happened (e.g., a dog barking), but exactly when it started and stopped. While deep learning has made massive strides here, real-world audio is messy. We often have to train models on "heterogeneous" datasets—data collected in different environments (scenarios) with varying levels of detail in their labels (granularities).

In a new paper, researchers from the Beijing University of Posts and Telecommunications introduce MTDA-HSED, a dual-branch architecture designed specifically to navigate these discrepancies. By combining a novel adapter-based tuning method with a sophisticated mid-fusion strategy, they’ve pushed the state-of-the-art for SED performance.

TL;DR

MTDA-HSED improves Sound Event Detection by using a "Mutual-Assistance" adapter to handle diverse recording scenarios and a "Dual-Branch Mid-Fusion" module to reconcile global and local audio features. The approach achieves a 5% improvement in mean partial AUC (mpAUC) over standard baselines on the DESED and MAESTRO datasets.

What Problem Does This Paper Address?

The authors identify two primary hurdles in modern SED:

  1. Scenario Discrepancy: Datasets are often a "mush" of different recording conditions. A model trained on clean studio audio might struggle with the noisy, reverberant environment of a smart city.
  2. Granularity Discrepancy: Some data comes with "hard" labels (exact timestamps), while others use "soft" labels (probabilities over time). Models often struggle to learn from both simultaneously without losing key information or getting "confused" by the differing levels of detail.

Existing methods usually rely on simple fine-tuning of pre-trained models (like BEATs), but these often fail to capture the nuances of complex, multi-scenario audio.

Methods

The MTDA-HSED architecture consists of two main innovations:

1. Mutual-Assistance Audio Adapter (M3A)

Instead of full fine-tuning, which is computationally expensive and can lead to forgetting, the authors use Parameter-Efficient Fine-Tuning (PEFT). They insert the M3A module into the Feed-Forward Networks (FFN) of a pre-trained BEATs (Audio Transformer) model.

  • The "Mutual" Part: It features a symmetrical structure with two paths. One path (Long-Term) uses an inverted bottleneck to capture "high-richness" information (the event plus its context). The other path (Short-Term) uses a standard bottleneck to focus on "low-richness" information (the event itself).
  • The Result: This allows the model to adapt to different temporal scales of sound events simultaneously.

2. Dual-Branch Mid-Fusion (DBMF)

Standard models often use a "late-fusion" approach, where different branches (like a Transformer for global context and a CNN for local patterns) only meet at the very end.

  • The Innovation: DBMF performs fusion during the encoding process. It uses a Multi-Head Cross-Attention mechanism where the global features from the Transformer act as the "Key" and "Value," and the local CNN features act as the "Query."
  • The Result: This ensures that local features are constantly informed by the broader acoustic context, preventing the loss of information that usually happens when granularities clash.

Datasets

The researchers evaluated their model using two staple benchmarks in the SED community, particularly relevant to the DCASE 2024 Task 4 challenge:

  • DESED: Focused on domestic environments, used primarily to evaluate performance with "hard" labels (PSDS1 metrics).
  • MAESTRO Real: Used to evaluate performance with "soft" labels (mpAUC metric).

Key Insights and Ideas

  • Symmetry is Key for Diversity: The ablation studies showed that having exactly two adapter paths (one for long-term, one for short-term) performed better than one or three. This suggests a "Goldilocks zone" for balancing temporal richness.
  • Projection Dimensions Matter: The authors found that scaling the feature dimension up by 4x for the Long-Term adapter and down to 1/4x for the Short-Term adapter yielded the best results. This drastic difference helps the model explicitly separate "context" from "content."
  • Mid-Fusion Prevents Confusion: By interacting every two blocks, the CNN and Transformer branches maintain their distinctive strengths while sharing enough information to resolve granularity conflicts.

Key Results

The MTDA-HSED model significantly outperformed the DCASE 2024 baseline and other popular PEFT methods:

  • mpAUC: Reached 0.757, a 5% absolute increase over the baseline (0.709).
  • PSDS1: Achieved 0.503, outperforming other additive methods like MONA (0.497) and ADAPTER (0.494).
  • Qualitative Evidence: Visualization of the feature maps showed that the Long-Term adapter successfully "filled in the gaps" between disjointed short-term sounds, effectively modeling the continuity of acoustic scenes.

Limitations and Future Work

While the paper demonstrates clear performance gains, it doesn't deeply explore the computational trade-offs of the cross-attention mechanism in the DBMF module, which could be a factor for real-time edge deployment.

Future work could involve testing this "Mutual-Assistance" philosophy on even more diverse backbones or extending the dual-branch approach to incorporate other modalities, like video or text descriptions of soundscapes.

Summary updated

BibTeX
@inproceedings{wang2025mtda,
  author = {Zehao Wang and Haobo Yue and Zhicheng Zhang and Da Mu and Jin Tang and Jianqin Yin},
  title = {{MTDA-HSED: Mutual-Assistance Tuning and Dual-Branch Aggregating for Heterogeneous Sound Event Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889194},
}
Figure