A Mixed Supervised Learning Framework For Target Sound Detection

Dongchao Yang, Helin Wang, Wenwu Wang, Yuexian Zou

Abstract

Target Sound Detection (TSD) is often hindered by the high cost of frame-level annotations. This paper proposes a mixed supervised learning framework that bridges the gap between expensive strong labels and abundant weak labels. Utilizing a "two-student" iterative learning strategy involving frame-level knowledge distillation and pseudo-supervised training, the framework achieves an 8% improvement in event-based F-scores over weakly supervised baselines, approaching the performance of fully supervised models while significantly reducing annotation labor.

Introduction

Target Sound Detection (TSD) involves identifying and localizing specific sound events within complex audio environments, guided by a reference clip. A major challenge in this field is the "supervision gap." Strongly supervised models require precise frame-level labels (start and end times), which are labor-intensive to produce. Conversely, weakly supervised models use clip-level labels, which are easier to obtain but often result in "boundary blindness," where the model fails to identify the exact onset and offset of a sound.

The Mixed Supervised Learning Framework

The proposed framework employs two "student" models that iteratively teach each other to leverage the strengths of different data types:

  1. The Strong Student ($f\_student$): Initially trained on a small, fully-annotated source dataset, this model excels at identifying temporal boundaries but lacks knowledge of the target domain's specific sounds.
  2. The Weak Student ($w\_student$): Trained on a large, weakly-annotated target dataset, it recognizes target sounds but lacks temporal precision.

The models interact through a mutual helping mechanism:

  • Knowledge Distillation (KD): The Strong Student transfers temporal boundary information to the Weak Student to improve its localization.
  • Pseudo-Supervised Training: The Weak Student generates "pseudo-labels" for the target domain, which the Strong Student uses for fine-tuning.
  • Adversarial Training: A discriminator ensures the feature extractor learns domain-invariant representations, mitigating domain mismatch between source and target datasets.

Key Insights

Boundary Blindness and Knowledge Distillation

Weakly supervised models typically identify the most distinct "peak" of an audio event but struggle with the edges. Knowledge distillation from a strongly-trained model helps "sharpen" these boundaries, allowing the model to capture the full duration of the event.

Comparison of predicted probabilities for Weak Supervision (WS) and Strong Supervision (SS).

Robustness to Label Noise

A critical finding of this research is that TSD models are remarkably resilient to errors in frame-level labels. Performance remains stable even with error rates as high as 35%, justifying the use of noisy pseudo-labels in the iterative training process.

The impact of label error rates on segment-based F-scores.

Results and Conclusion

The Mixed Supervision (MS) method achieved an event-based F-score of 47.56% on the L-TSD test set, a significant increase from the 39.07% achieved by standard weak supervision. This performance closely rivals the 50.4% achieved by full strong supervision, demonstrating that the framework effectively leverages weak labels to reduce annotation costs without sacrificing significant accuracy. Future work will explore applying this mixed-supervision logic to general Sound Event Detection (SED) and computer vision tasks.

Summary updated

Abstract

Target sound detection (TSD) aims to detect the target sound from mixture audio given the reference information. Previous works have shown that TSD models can be trained on fully-annotated (frame-level label) or weakly-annotated (clip-level label) data. However, there are some clear evidences show that the performance of the model trained on weakly-annotated data is worse than that trained on fully-annotated data. To fill this gap, we provide a mixed supervision perspective, in which learning novel categories (target domain) using weak annotations with the help of full annotations of existing base categories (source domain). To realize this, a mixed supervised learning framework is proposed, which contains two mutually-helping student models (\textit{f\_student} and \textit{w\_student}) that learn from fully-annotated and weakly-annotated data, respectively. The motivation is that \textit{f\_student} learned from fully-annotated data has a better ability to capture detailed information than \textit{w\_student}. Thus, we first let \textit{f\_student} guide \textit{w\_student} to learn the ability to capture details, so \textit{w\_student} can perform better in the target domain. Then we let \textit{w\_student} guide \textit{f\_student} to fine-tune on the target domain. The process can be repeated several times so that the two students perform very well in the target domain. To evaluate our method, we built three TSD datasets based on UrbanSound and Audioset. Experimental results show that our methods offer about 8\% improvement in event-based F-score as compared with a recent baseline.
BibTeX
@inproceedings{yang2022mixed,
  author = {Dongchao Yang and Helin Wang and Wenwu Wang and Yuexian Zou},
  title = {{A Mixed Supervised Learning Framework For Target Sound Detection}},
  booktitle = {DCASE},
  year = {2022},
  url = {https://openreview.net/forum?id=FJffTWPKaF},
}
Figure