TL;DR
This paper tackles the challenge of imbalanced learning between audio and visual modalities in weakly-supervised audio-visual video parsing (WS-AVVP). They introduce a Dynamic Gradient Modulation (DGM) mechanism to balance feature learning and a Modality-Separated Decision Unit (MSDU) for more accurate imbalance measurement, achieving state-of-the-art results.
What Problem Does This Paper Address?
Weakly-supervised audio-visual video parsing (WS-AVVP) aims to identify the start and end times of events in videos and their categories, using only video-level labels for training. While existing methods leverage both audio and visual information, they often overlook a crucial issue: the inherent imbalance in how well each modality contributes to learning. For instance, a "basketball bounce" might be visually obvious but harder to discern from audio alone. This leads to one modality dominating the training process, hindering the learning of the weaker modality. This imbalance can result in uncoordinated convergence and suboptimal overall performance. The paper addresses this by proposing methods to explicitly balance the learning of audio and visual features.
Methods
The core of the proposed method lies in two key components:
- Dynamic Gradient Modulation (DGM): This mechanism aims to balance the feature learning between audio and visual modalities. It works by:
- Imbalance Metric: Designing a novel metric to quantify the degree of imbalanced feature learning. This metric considers both the predicted scores for correct categories and the global prediction distribution across all categories, making it more comprehensive than previous approaches.
- Gradient Modulation: Using the calculated imbalance degree to modulate the gradients of the modality-specific feature encoders. This effectively directs more optimization attention towards the underperforming modality.
- Modality-Separated Decision Unit (MSDU): The authors observed that the cross-attention mechanism, commonly used in WS-AVVP, can lead to "confusing" multimodal features. This makes it difficult to accurately measure the imbalance between pure audio and visual features. To overcome this, MSDU is introduced. It's embedded before the cross-attention block and uses separate attention and classification modules for audio and visual features to obtain pure, modality-specific predictions. These pure predictions are then used for imbalance measurement, leading to a more precise assessment and enhancing the effectiveness of DGM.
Datasets
The proposed method is evaluated on the following datasets:
- LLP: A large-scale dataset for WS-AVVP, containing 11,849 YouTube video clips of 10-second duration across 25 semantic categories.
- CREMA-D: A dataset for speech emotion recognition, featuring 7,442 video clips from 91 actors.
- AVE: A benchmark for audio-visual event localization, with 4,143 10-second videos and 28 event categories.
Key Insights and Ideas
- Multimodal Imbalance is a Real Problem: The paper convincingly argues that the unequal contribution of audio and visual modalities is a significant bottleneck in WS-AVVP, leading to suboptimal learning.
- Beyond Simple Score Ratios: The proposed imbalance metric is more sophisticated by considering the entire prediction distribution, not just scores for the correct class, leading to a more robust measure.
- Separation is Key for Accurate Measurement: The MSDU's insight that confusing multimodal features hinder accurate imbalance assessment is crucial. Separating modality processing before fusion allows for a purer measurement.
- DGM and MSDU are Synergistic: The combination of DGM and MSDU yields significantly better results than either component alone, highlighting their complementary nature.
- Generalizability: The DGM mechanism can be effectively integrated with different existing WS-AVVP architectures (e.g., HAN, JoMoLD), demonstrating its broad applicability.
Key Results
The proposed method achieves state-of-the-art performance on the LLP dataset, outperforming recent methods like JoMoLD by an average of 1.48% on segment-level metrics and 1.44% on event-level metrics. Similar performance improvements were observed on the CREMA-D and AVE datasets, demonstrating the method's effectiveness and generalization capabilities. Ablation studies show that DGM alone provides improvements, but the combination with MSDU leads to substantial gains (e.g., an additional 1.14% and 0.94% improvement on segment-level and event-level metrics respectively when compared to "JoMoLD+DGM"). Qualitative results also show improved localization of events, especially in weaker modalities.
Limitations and Future Work
- Limitations: The paper notes that while DGM generally improves performance, the audio-visual sub-task performance might drop slightly when using the "score" strategy alone for imbalance measurement, attributed to confusing multimodal features. This is addressed by the MSDU.
- Future Work: The paper implicitly suggests that further exploration of more sophisticated imbalance metrics and fusion strategies could lead to even greater improvements. The generalizability of DGM to other multimodal tasks beyond WS-AVVP is also a promising avenue.
Summary updated