Fine-tuning Audio Spectrogram Transformer with Task-aware Adapters for Sound Event Detection

Shuo Li, Yan Song, Ian McLoughlin, Lin Liu, Jin Li, Li-Rong Dai2023 · DOI

TL;DR

This paper introduces a novel "task-aware fine-tuning" approach to adapt a pre-trained audio transformer (PaSST) for Sound Event Detection (SED). By using specialized adapters for both SED and Audio Tagging (AT) tasks, and strategically applying them to different layers of the transformer, they achieve state-of-the-art performance. They also propose a self-distilled mean teacher method to further improve robustness against noisy labels.

What Problem Does This Paper Address?

Sound Event Detection (SED) is crucial for understanding audio content, identifying specific sounds and their timings within a clip. However, acquiring large, precisely labeled datasets for SED is challenging and expensive. This has led to a focus on weakly-supervised SED, where only clip-level labels are available. While pre-trained models like the Audio Spectrogram Transformer (AST) and its successor PaSST show promise, simply fine-tuning them on SED tasks often fails to fully leverage their capabilities. Specifically, existing methods tend to fine-tune from a single layer, which may not optimally exploit both the local acoustic details and the broader semantic context needed for accurate SED. This paper aims to bridge this gap by developing a more sophisticated fine-tuning strategy.

Methods

The core of this work lies in two main innovations:

  • Task-Aware Adapters and Fine-Tuning: Instead of a one-size-fits-all fine-tuning approach, the authors introduce "task-aware adapters."
    • SED-adapter: Designed to capture local acoustic information, this adapter is attached to shallower layers of the PaSST model.
  • AT-adapter: Focused on extracting global semantic information, this adapter is applied to deeper layers.
  • Task-Aware Fine-Tuning: This strategy combines both adapters, allowing the model to learn from both local and semantic features simultaneously by fine-tuning PaSST across multiple layers. This is a departure from previous methods that typically fine-tuned from a single layer.
  • Self-Distillated Mean Teacher (SdMT): To combat the inherent noise in weakly-labeled data (e.g., inaccurate timestamps), they propose SdMT. This method uses a "teacher" model (an exponential moving average of a "main-student" model) to generate soft pseudo-labels. A "vice-student" model is then trained using these soft labels via knowledge distillation. This is contrasted with hard distillation, which uses thresholded labels.

Datasets

The experiments were conducted on the DCASE2022 task4 development set. This dataset includes:

  • Weakly-labeled clips
  • Strongly-labeled clips
  • Synthetic strongly-labeled clips
  • Unlabeled in-domain clips

The validation set consists of strongly-labeled clips. The authors do not introduce a new benchmark dataset.

Key Insights and Ideas

  • Layer-Specific Feature Exploitation: The paper highlights that different layers of a pre-trained transformer capture distinct types of information. Shallower layers are better for local acoustic features crucial for SED, while deeper layers excel at capturing semantic context relevant for AT. This insight directly motivates the task-aware adapter placement.
  • The Power of Combined Task Learning: By using both SED and AT adapters on different layers, the model can learn to perform both tasks concurrently during fine-tuning. This synergistic approach leads to better SED performance than solely focusing on SED.
  • Soft Distillation for Robustness: Soft pseudo-labels generated by the teacher model are more informative and less prone to noise than hard labels. This allows the vice-student to learn more effectively and robustly, especially when dealing with potentially inaccurate annotations.
  • PaSST's Strong Foundation: The success of the approach underscores the effectiveness of PaSST, a pre-trained transformer model, as a powerful backbone for downstream audio tasks like SED.

Key Results

The proposed methods achieved state-of-the-art performance on the DCASE2022 task4 development set:

  • EB-F1 (Event-Based F1-score): 64.85%
  • PSDS1 (Polyphonic Sound Detection Score scenario1): 0.5548

These results significantly outperform previous state-of-the-art systems, including AST-SED (59.60% EB-F1, 0.5140 PSDS1). The task-aware fine-tuning and SdMT methods individually contribute to performance gains, with their combination yielding the best results. Soft distillation in SdMT also showed superior performance compared to hard distillation.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the "Independent" teacher in task-aware fine-tuning (where teacher adapters guide student adapters independently) outperforms the baseline, its full potential might not have been explored.
  • The performance of hard distillation in SdMT was limited, possibly due to the difficulty of determining events and the sub-optimality of a fixed threshold (0.5) for generating hard labels.
  • Future Work:
    • The authors aim to explore more effective fine-tuning methods to better leverage pre-trained AT models.
  • They plan to further investigate how to effectively train SED systems using soft labels.
  • Investigating the "Independent" teacher configuration in more depth is also a potential future direction.

Summary updated

Abstract

<p dir="ltr">In this paper, we present a task-aware fine-tuning method to transfer Patchout faSt Spectrogram Transformer (PaSST) model to sound event detection (SED) task. Pretrained PaSST has shown significant performance on audio tagging (AT) and SED tasks, but it is not optimal to fine-tune the model from a single layer as the local and semantic information have not been well exploited. To address this, we first introduce task-aware adapters including SED-adapter and AT-adapter to fine-tune PaSST for SED and AT task respectively, and then propose task-aware fine-tuning to combine local information from shallower layer with semantic information from deeper layer, based on task-aware adapters. Besides, we propose the self-distillated mean teacher (SdMT) to train a robust student model with soft pseudo labels from teacher. Experiments are conducted on DCASE2022 task4 development set, the EB-F1 of 64.85% and PSDS1 of 0.5548 are achieved which outperform previous state-of-the-art systems.</p>
BibTeX
@misc{li2023fine,
  author = {Shuo Li and Yan Song and Ian McLoughlin and Lin Liu and Jin Li and Li-Rong Dai},
  title = {{Fine-tuning Audio Spectrogram Transformer with Task-aware Adapters for Sound Event Detection}},
  year = {2023},
  doi = {10.21437/interspeech.2023-1174},
}
Figure