Abstract
Weakly-supervised video moment retrieval (wsVMR) aims to localize specific events in videos using only video-level text descriptions during training. A significant challenge in current wsVMR systems is the "scene-proposal mismatch," where the number of generated candidate segments (proposals) does not align with the actual density of events in a video. We present SCANet (Scene Complexity Aware Network), which adaptively adjusts proposal generation based on a video's estimated scene complexity. By leveraging query diversity to gauge event density, SCANet achieves state-of-the-art performance on Charades-STA, ActivityNet Captions, and TV Show Retrieval benchmarks.
The Scene-Proposal Mismatch Problem
In Video Moment Retrieval (VMR), the goal is to identify the temporal boundaries of an event described by a text query. Because manual timestamp annotation is labor-intensive, wsVMR models learn from coarse video-level associations. However, existing models typically generate a fixed number of proposals or scale them linearly with video length.
As shown in Figure 1, this leads to a fundamental mismatch: a long video (e.g., 160 seconds) might contain only one continuous action, yet the system generates excessive proposals. Conversely, a short, dense video (e.g., 17 seconds) might contain multiple distinct actions but receive too few proposals. The authors found that this mismatch affects approximately 41% of videos in standard benchmarks, leading to "irregular retrieval quality" where performance drops significantly when the proposal count is poorly matched to the scene count.
The SCANet Framework
SCANet introduces a three-pronged approach to mitigate this mismatch by making the network aware of scene complexity:
1. Scene Complexity Estimation
Since ground-truth scene counts are unavailable in a weakly-supervised setting, SCANet uses the diversity of text queries associated with a video as a proxy. By extracting nouns from all queries for a given video and removing semantic redundancies (e.g., grouping "food" and "bread"), the model estimates the number of unique scenes ($\alpha$).
2. Complexity-Adaptive Proposal Generation (CPG)
Rather than using a static heuristic, SCANet employs a learnable codebook and a Gumbel-Softmax layer to dynamically determine the number of proposals based on the estimated complexity $\alpha$. This ensures that complex videos receive more granular candidates while simple videos are processed with fewer, broader proposals. Additionally, a Flatten Gaussian Mask is used to ensure the model attends to features across the entire duration of a segment uniformly.
3. Complexity-Adaptive Proposal Enhancement (CPE)
To further refine the model's discriminative power, SCANet incorporates two self-supervised objectives:
- Cross-modal Reconstruction: The model learns to predict masked words in queries or masked frames in videos, strengthening the alignment between visual and textual features.
- Hierarchical Contrastive Learning: The system is trained to distinguish target moments from "hard negatives"—semantically similar scenes—both within the same video and across the entire dataset (corpus-level).
Experimental Results
SCANet was evaluated on three major benchmarks: Charades-STA, ActivityNet Captions, and TV Show Retrieval (TVR). It consistently outperformed previous state-of-the-art (SOTA) methods:
- Charades-STA: Achieved an R@1 (IoU=0.5) of 50.85%, surpassing the previous best (CPL) at 49.05%.
- ActivityNet Captions: Reached an R@1 (IoU=0.5) of 31.52%, demonstrating effectiveness on longer, more diverse videos.
- TVR: On this large-scale dataset, SCANet showed a dramatic improvement, reaching 37.51% (R@1, IoU=0.1) compared to the 17.61% baseline.
Qualitative analysis confirms that SCANet successfully adapts its proposal density, accurately capturing rapid-fire events in short clips and singular events in long clips.
Conclusion and Future Work
SCANet demonstrates that video length is an insufficient metric for determining proposal density in VMR. By introducing scene complexity as a guiding factor, the model significantly improves retrieval accuracy. While the current implementation relies on multiple queries per video for training, future work involves refining "Scene Complexity Neural Estimators" to predict complexity directly from raw pixels, making the system even more robust for real-world, single-query applications.
Summary updated