Exploring Heterogeneous Clues for Weakly-Supervised Audio-Visual Video Parsing

Yu Wu, Yi YangCVPR 2021 · OpenReview

Abstract

We investigate the weakly-supervised audio-visual video parsing task, which aims to parse a video into temporal event segments and predict the audible or visible event categories. The task is challenging since there only exist video-level event labels for training, without indicating the temporal boundaries and modalities. Previous works take the overall event labels to supervise both audio and visual model predictions. However, we argue that such overall labels harm the model training due to the audio-visual asynchrony. In this paper, we tackle this issue by leveraging the cross-modal correspondence of audio and visual signals. We generate reliable event labels individually for each modality by swapping audio and visual tracks with other unrelated videos. If the original visual/audio data contain event clues, the event prediction from the newly assembled data would still be highly confident. In addition, we propose the cross-modal audio-visual contrastive learning to induce temporal difference on attention models within videos. Experiments show we outperform state-of-the-art methods by a large margin.

The Challenge of Audio-Visual Asynchrony

In an ideal scenario, audio and visual signals in a video are perfectly synchronized—for instance, seeing a dog bark while simultaneously hearing the sound. However, real-world video is often asynchronous. A sports commentator might discuss a basketball game while the camera shows a crowd shot, or a player might be interviewed off-court while game highlights are shown.

Figure 1: An illustration of audio-visual asynchrony in a basketball video, where visual events and audio events do not perfectly overlap in time or modality.

This "audio-visual asynchrony" poses a significant hurdle for weakly-supervised parsing, where only video-level labels (e.g., "Basketball" and "Speech") are available. Standard models often assume these labels apply equally to both modalities at all times, leading to "noisy" training where the model tries to find "Speech" in visual frames where no one is talking.

Methodology: Probing Modalities via Track Swapping

To address modality uncertainty, we propose Modality-Aware Label Refining (MA). This method uses a "swap test" to determine if an event is truly present in a specific modality. By taking a target video and swapping its audio track with that of an unrelated video, we can probe the model's confidence.

Figure 6: The proposed track-swapping mechanism. By assembling new inputs with unrelated audio/visual tracks, the model can identify which events are modality-specific.

If the model maintains high confidence for a "Basketball" prediction when using the original visual track paired with a new, unrelated audio track (like a dog barking), it provides a strong signal that "Basketball" is a visual event. This allows us to generate refined, modality-specific pseudo-labels that protect the model from being misled by ambiguous global labels.

Furthermore, we introduce Cross-Modal Contrastive Learning to sharpen temporal boundaries. Standard attention mechanisms often blur features across the entire video duration. By forcing the model to align aggregated audio features of a specific second with the corresponding visual features of that same second—while pushing them away from visual features of different seconds—we encourage the model to capture precise temporal differences.

Experimental Results

We evaluated our approach on the Look, Listen and Parse (LLP) dataset, which contains over 11,000 YouTube clips across 25 categories. Our model, dubbed MA, established a new state-of-the-art performance:

  • Overall Audio-Visual Parsing: Improved segment-level F-score from 48.9% to 55.1% over the previous best method (HAN).
  • Visual Event Parsing: Achieved a significant jump from 52.9% to 60.0%, confirming that visual labels were previously the most affected by asynchrony noise.
  • Audio Event Parsing: Showed steady gains (60.3%), reflecting the naturally higher consistency of audio signals with global labels in this dataset.

Conclusion

By treating audio and visual modalities as independent but related witnesses, our track-swapping approach effectively mitigates the noise introduced by asynchrony in weakly-supervised settings. Coupled with contrastive learning to regularize temporal attention, this method provides a robust framework for parsing complex, real-world video data into its constituent audible and visible parts.

Summary updated

Abstract

We investigate the weakly-supervised audio-visual video parsing task, which aims to parse a video into temporal event segments and predict the audible or visible event categories. The task is challenging since there only exist video-level event labels for training, without indicating the temporal boundaries and modalities. Previous works take the overall event labels to supervise both audio and visual model predictions. However, we argue that such overall labels harm the model training due to the audio-visual asynchrony. For example, commentators speak in a basketball video, but we cannot visually find the speakers. In this paper, we tackle this issue by leveraging the cross-modal correspondence of audio and visual signals. We generate reliable event labels individually for each modality by swapping audio and visual tracks with other unrelated videos. If the original visual/audio data contain event clues, the event prediction from the newly assembled data would still be highly confident. In this way, we could protect our models from being misled by ambiguous event labels. In addition, we propose the cross-modal audio-visual contrastive learning to induce temporal difference on attention models within videos, i.e., urging the model to pick the current temporal segment from all context candidates. Experiments show we outperform state-of-the-art methods by a large margin.
BibTeX
@inproceedings{wu2021exploring,
  author = {Yu Wu and Yi Yang},
  title = {{Exploring Heterogeneous Clues for Weakly-Supervised Audio-Visual Video Parsing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=N7jzbH1xQN},
}
Figure