Introduction
Traditional video retrieval research has primarily focused on Text-to-Video Retrieval (T2VR), where a short text query describes the entirety of a brief video clip. However, real-world search scenarios often involve Partially Relevant Video Retrieval (PRVR). In PRVR, a specific query (e.g., "a person drinking water") may only correspond to a small segment within a much longer video. Existing methods for this task often rely on static, pre-extracted features and fail to capture the nuanced relationships between different temporal segments.
The Challenge of Partial Relevance
To effectively retrieve partially relevant segments, a model must understand two critical types of relationships:
- Intra-segment relations: Recognizing how high-level activities (e.g., "eating dinner") are composed of discrete sub-actions (e.g., "putting down a glass").
- Inter-segment relations: Understanding how the meaning of a segment is influenced by preceding or succeeding events (e.g., the word "again" in "drinks again" implies a prior drinking event).
As shown in Figure 1, standard models often struggle with these dependencies because they lack a structured way to model the video's temporal hierarchy.
Methodology: The ProPy Framework
The authors propose ProPy (Interactive Prompt Pyramid), a framework that adapts the CLIP vision-language model for PRVR through a hierarchical and interactive approach. ProPy introduces three core components:
1. Interactive Prompt Pyramid
Instead of representing a video with a single global token, ProPy utilizes a hierarchy of learnable "event prompts." The pyramid's base layer captures fine-grained, short-duration segments, while higher layers aggregate these into progressively longer, more abstract temporal spans. This allows the model to process the video at multiple granularities simultaneously.
2. Ancestor-Descendant (A-D) Interaction
To facilitate communication between prompts without introducing excessive noise, ProPy employs a sparse attention mechanism called Ancestor-Descendant Interaction. A prompt is only permitted to attend to its "ancestors" (larger segments that contain it) and its "descendants" (smaller segments it encompasses). This structured communication captures hierarchical context while preventing "feature leakage" from unrelated temporal segments.
3. Temporal Adapters
Because the base CLIP model is designed for static images, ProPy incorporates lightweight 3D-CNN temporal adapters into the visual branch. These modules enable the model to capture motion and temporal flow, which are essential for distinguishing complex actions.
Experimental Evaluation
ProPy was evaluated on four major benchmarks: TVR, ActivityNet Captions, Charades-STA, and QVHighlights. The model achieved new state-of-the-art (SOTA) performance across all datasets, often by substantial margins:
- TVR: Achieved a 7.7% absolute improvement in Recall@1 (R@1) over previous leading models.
- ActivityNet Captions: Demonstrated a 7.2% absolute improvement in R@1, proving effective for long-form videos (averaging 118 seconds).
- QVHighlights: Reached 37.4% R@1, significantly outperforming existing competitors in moment retrieval and highlight detection.
Qualitative analysis confirms that the model successfully utilizes hierarchical context. For instance, when queried with "hands the book back," the model leverages the "ancestor" context of an earlier "opening a book" segment to correctly identify the target moment.
Discussion and Conclusion
ProPy demonstrates that structured, hierarchical communication is more effective for video understanding than dense, all-to-all attention. By freezing approximately 95% of CLIP’s parameters and only training the prompts and adapters, the model remains relatively efficient.
Future research may explore adaptive pyramid structures, where the model dynamically determines segment lengths based on video content rather than relying on fixed hyperparameters. Additionally, more sophisticated temporal sampling strategies could further enhance performance for extremely long-duration videos where memory overhead remains a challenge.
Summary updated