MTDA-HSED: Mutual-Assistance Tuning and Dual-Branch Aggregating for Heterogeneous Sound Event Detection

Zehao Wang, Haobo Yue, Zhicheng Zhang, Da Mu, Jin Tang, Jianqin Yin

Sound Event Detection (SED) involves identifying the occurrence and precise temporal boundaries of sound events in audio streams. Real-world audio data often presents "heterogeneous" challenges, characterized by scenario discrepancies (varying recording environments) and granularity discrepancies (varying label precision). To address these, the MTDA-HSED framework introduces a dual-branch architecture that leverages parameter-efficient fine-tuning and mid-fusion strategies.

Methodology

The MTDA-HSED architecture is designed to bridge the gap between global context and local acoustic patterns while adapting to diverse data distributions.

The overall architecture of MTDA-HSED (right) alongside the detailed BEATs Block featuring the M3A-FFN (left).

Mutual-Assistance Audio Adapter (M3A)

Instead of full fine-tuning of a pre-trained BEATs model, MTDA-HSED utilizes the Mutual-Assistance Audio Adapter (M3A) within the Feed-Forward Networks (FFN). The M3A module employs a symmetrical dual-path structure:

  • Long-Term Path: Uses an inverted bottleneck to capture high-richness information, including the sound event and its surrounding context.
  • Short-Term Path: Uses a standard bottleneck to focus on low-richness information, isolating the event itself.

This "mutual-assistance" allows the model to simultaneously adapt to different temporal scales, effectively handling scenario discrepancies.

Dual-Branch Mid-Fusion (DBMF)

To reconcile granularity discrepancies, the model employs a Dual-Branch Mid-Fusion (DBMF) module. Unlike late-fusion approaches, DBMF facilitates interaction between the global Transformer branch and the local CNN branch during the encoding process.

Detailed view of the Dual-Branch Mid-Fusion (DBMF) block utilizing Multi-Head Cross-Attention.

As shown in the figure above, the DBMF block uses a Multi-Head Cross-Attention mechanism. Global features from the Transformer serve as the "Key" and "Value," while local CNN features serve as the "Query." This ensures that local feature extraction is continuously informed by the broader acoustic context, preserving critical information across different label granularities.

Experimental Results

The model was evaluated on the DESED and MAESTRO Real datasets, benchmarks central to the DCASE 2024 Task 4 challenge.

  • Performance Metrics: MTDA-HSED achieved a mean partial AUC (mpAUC) of 0.757, representing a 5% absolute improvement over the DCASE 2024 baseline (0.709).
  • SED Accuracy: On the DESED dataset, it reached a PSDS1 score of 0.503, outperforming other popular parameter-efficient fine-tuning (PEFT) methods such as MONA and standard ADAPTER.
  • Ablation Studies: Results indicated that a symmetrical dual-path adapter (one long-term, one short-term) is optimal. Specifically, scaling feature dimensions by 4x for the long-term path and 1/4x for the short-term path provided the best balance between context and content.

Conclusion

MTDA-HSED demonstrates that combining symmetrical adapter-based tuning with mid-fusion cross-attention significantly enhances performance in heterogeneous sound event detection. By explicitly modeling both long-term context and short-term event details, the architecture effectively navigates the complexities of real-world audio datasets. Future research may explore the scalability of this approach to other multi-modal tasks or more diverse pre-trained backbones.

Summary updated

BibTeX
@inproceedings{wang2025mtda,
  author = {Zehao Wang and Haobo Yue and Zhicheng Zhang and Da Mu and Jin Tang and Jianqin Yin},
  title = {{MTDA-HSED: Mutual-Assistance Tuning and Dual-Branch Aggregating for Heterogeneous Sound Event Detection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10889194},
}
Figure