Solution for Temporal Sound Localisation Task of ECCV Second Perception Test Challenge 2024

Haowei Gu, Weihao Zhu, Yang YangarXiv 2024 · DOI · arXiv

TL;DR

By prioritizing audio features over visual ones, researchers from the Nanjing University of Science and Technology secured first place in the ECCV 2024 Temporal Sound Localisation (TSL) challenge. Their winning strategy involved a multi-model ensemble for audio feature extraction and a modified Weighted Boxes Fusion (WBF) post-processing step, achieving a top score of 0.4925 mAP.

What Problem Does This Paper Address?

Temporal Sound Localisation (TSL) is the task of identifying exactly when a specific sound occurs in a video and classifying what that sound is (e.g., a dog barking or a glass breaking). While it sounds straightforward, it is a significant challenge in multimodal video understanding.

The authors identified a key limitation in previous state-of-the-art approaches: many models treat audio and video modalities with equal importance. However, since the goal is specifically to localize sound events, the authors hypothesized that the audio stream contains more critical information than the visual stream. Their work focuses on proving this "audio-first" hypothesis and engineering a system that extracts much richer acoustic representations to improve localization accuracy.

Methods

The team built their solution on top of Actionformer, a popular foundational model for temporal action localization that uses a Transformer-based encoder and a multi-scale feature pyramid.

The core innovations lie in their multimodal feature extraction and fusion strategy:

  • Enhanced Audio Features: Instead of relying on a single audio model, they used a "multi-model strategy." They extracted features using three high-performing pre-trained models:
  • BEATS: An iterative audio pre-training model.
  • CAV-MAE (variant 1): Fine-tuned on the AudioSet dataset.
  • CAV-MAE (variant 2): Fine-tuned on the VGGSound dataset. These features (each 768-dimensional) were concatenated into a massive 2304-dimensional vector to capture a wide range of acoustic characteristics.

  • Visual Features: They used fine-tuned versions of UMT-Large and VideoMAE-Large to provide the visual context.

  • Early Fusion: Video and audio features were temporally aligned and then concatenated along the channel dimension before being fed into the Actionformer backbone.
  • Post-Processing (Modified WBF): They adapted Weighted Boxes Fusion (WBF)—a technique typically used in 2D object detection to merge bounding boxes—to work for 1D temporal segments. They ensembled predictions from models trained for different durations (20, 25, and 30 epochs) to produce a more robust final output.

Datasets

The study utilized the dataset provided for the Perception Test Challenge 2024 (Track 3).

  • Content: High-resolution videos (RGB + audio).
  • Classes: 17 distinct sound event categories.
  • Constraints: The authors relied strictly on the official training data provided by the competition organizers, without using external data for training.

Key Insights and Ideas

  • Audio is King for TSL: The most significant "aha moment" was the experimental validation that audio features are more effective than video features for this specific task. In their ablation study, using only audio features (0.4560 mAP) significantly outperformed using only video features (0.4102 mAP).
  • Diversity in Pre-training Matters: By using CAV-MAE models fine-tuned on different datasets (AudioSet vs. VGGSound), the model could leverage different types of acoustic "knowledge," leading to a more comprehensive representation than a single model could provide.
  • Temporal Ensembling: Applying WBF to temporal segments from different training checkpoints proved to be a highly effective way to squeeze extra performance out of the model, helping it rank first on the leaderboard.

Key Results

The proposed method achieved a mean Average Precision (mAP) of 0.4925 on the final test set, ranking 1st in the competition.

Model mAP
Baseline (2023 Champion) 0.3925
VideoMAE (Video only) 0.4102
CAV-MAE (Sound only) 0.4560
Proposed Method (All + WBF) 0.4925

The results clearly show that while multimodal fusion is best, the "Sound only" approach is surprisingly potent, outperforming the previous year's champion baseline by a wide margin.

Limitations and Future Work

The paper focuses primarily on the competition-winning engineering strategy. While highly effective, the authors do not deeply explore the computational overhead of running five large-scale pre-trained models (UMT, VideoMAE, BEATS, and two CAV-MAEs) simultaneously for feature extraction.

Future work might involve investigating more efficient ways to achieve similar results—perhaps through knowledge distillation—or exploring "late fusion" techniques to see if they offer further improvements over the "early fusion" approach used here.

Summary updated

Abstract

This report proposes an improved method for the Temporal Sound Localisation (TSL) task, which localizes and classifies the sound events occurring in the video according to a predefined set of sound classes. The champion solution from last year's first competition has explored the TSL by fusing audio and video modalities with the same weight. Considering the TSL task aims to localize sound events, we conduct relevant experiments that demonstrated the superiority of sound features (Section 3). Based on our findings, to enhance audio modality features, we employ various models to extract audio features, such as InterVideo, CaVMAE, and VideoMAE models. Our approach ranks first in the final test with a score of 0.4925.
BibTeX
@misc{gu2024solution,
  author = {Haowei Gu and Weihao Zhu and Yang Yang},
  title = {{Solution for Temporal Sound Localisation Task of ECCV Second Perception Test Challenge 2024}},
  year = {2024},
  eprint = {2409.19595},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.19595},
}
Figure