SCANet: Scene Complexity Aware Network for Weakly-Supervised Video Moment Retrieval

Sunjae Yoon, Gwanhyeong Koo, Dahyun Kim, Chang D. Yoo

Abstract

Weakly-supervised video moment retrieval (wsVMR) aims to localize specific events in videos using only video-level text descriptions during training. A significant challenge in current wsVMR systems is the "scene-proposal mismatch," where the number of generated candidate segments (proposals) does not align with the actual density of events in a video. We present SCANet (Scene Complexity Aware Network), which adaptively adjusts proposal generation based on a video's estimated scene complexity. By leveraging query diversity to gauge event density, SCANet achieves state-of-the-art performance on Charades-STA, ActivityNet Captions, and TV Show Retrieval benchmarks.

The Scene-Proposal Mismatch Problem

In Video Moment Retrieval (VMR), the goal is to identify the temporal boundaries of an event described by a text query. Because manual timestamp annotation is labor-intensive, wsVMR models learn from coarse video-level associations. However, existing models typically generate a fixed number of proposals or scale them linearly with video length.

Figure 1: (a) Illustration of scene-proposal mismatch where long videos may have few scenes and short videos many. (b) Heatmap showing irregular retrieval quality when proposal counts don't match scene counts. (c) The proposed method for estimating scene complexity from queries.

As shown in Figure 1, this leads to a fundamental mismatch: a long video (e.g., 160 seconds) might contain only one continuous action, yet the system generates excessive proposals. Conversely, a short, dense video (e.g., 17 seconds) might contain multiple distinct actions but receive too few proposals. The authors found that this mismatch affects approximately 41% of videos in standard benchmarks, leading to "irregular retrieval quality" where performance drops significantly when the proposal count is poorly matched to the scene count.

The SCANet Framework

SCANet introduces a three-pronged approach to mitigate this mismatch by making the network aware of scene complexity:

1. Scene Complexity Estimation

Since ground-truth scene counts are unavailable in a weakly-supervised setting, SCANet uses the diversity of text queries associated with a video as a proxy. By extracting nouns from all queries for a given video and removing semantic redundancies (e.g., grouping "food" and "bread"), the model estimates the number of unique scenes ($\alpha$).

2. Complexity-Adaptive Proposal Generation (CPG)

Rather than using a static heuristic, SCANet employs a learnable codebook and a Gumbel-Softmax layer to dynamically determine the number of proposals based on the estimated complexity $\alpha$. This ensures that complex videos receive more granular candidates while simple videos are processed with fewer, broader proposals. Additionally, a Flatten Gaussian Mask is used to ensure the model attends to features across the entire duration of a segment uniformly.

3. Complexity-Adaptive Proposal Enhancement (CPE)

To further refine the model's discriminative power, SCANet incorporates two self-supervised objectives:

  • Cross-modal Reconstruction: The model learns to predict masked words in queries or masked frames in videos, strengthening the alignment between visual and textual features.
  • Hierarchical Contrastive Learning: The system is trained to distinguish target moments from "hard negatives"—semantically similar scenes—both within the same video and across the entire dataset (corpus-level).

Experimental Results

SCANet was evaluated on three major benchmarks: Charades-STA, ActivityNet Captions, and TV Show Retrieval (TVR). It consistently outperformed previous state-of-the-art (SOTA) methods:

  • Charades-STA: Achieved an R@1 (IoU=0.5) of 50.85%, surpassing the previous best (CPL) at 49.05%.
  • ActivityNet Captions: Reached an R@1 (IoU=0.5) of 31.52%, demonstrating effectiveness on longer, more diverse videos.
  • TVR: On this large-scale dataset, SCANet showed a dramatic improvement, reaching 37.51% (R@1, IoU=0.1) compared to the 17.61% baseline.

Qualitative analysis confirms that SCANet successfully adapts its proposal density, accurately capturing rapid-fire events in short clips and singular events in long clips.

Conclusion and Future Work

SCANet demonstrates that video length is an insufficient metric for determining proposal density in VMR. By introducing scene complexity as a guiding factor, the model significantly improves retrieval accuracy. While the current implementation relies on multiple queries per video for training, future work involves refining "Scene Complexity Neural Estimators" to predict complexity directly from raw pixels, making the system even more robust for real-world, single-query applications.

Summary updated

Abstract

Video moment retrieval aims to localize moments in video corresponding to a given language query. To avoid the expensive cost of annotating the temporal moments, weakly-supervised VMR (wsVMR) systems have been studied. For such systems, generating a number of proposals as moment candidates and then selecting the most appropriate proposal has been a popular approach. These proposals are assumed to contain many distinguishable scenes in a video as candidates. However, existing proposals of wsVMR systems do not respect the varying numbers of scenes in each video, where the proposals are heuristically determined irrespective of the video. We argue that the retrieval system should be able to counter the complexities caused by varying numbers of scenes in each video. To this end, we present a novel concept of a retrieval system referred to as Scene Complexity Aware Network (SCANet), which measures the `scene complexity' of multiple scenes in each video and generates adaptive proposals responding to variable complexities of scenes in each video. Experimental results on three retrieval benchmarks (i.e., Charades-STA, ActivityNet, TVR) achieve state-of-the-art performances and demonstrate the effectiveness of incorporating the scene complexity.
BibTeX
@inproceedings{yoon2023scanet,
  author = {Sunjae Yoon and Gwanhyeong Koo and Dahyun Kim and Chang D. Yoo},
  title = {{SCANet: Scene Complexity Aware Network for Weakly-Supervised Video Moment Retrieval}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01249},
}
Figure