TL;DR
ProPy introduces a hierarchical "Prompt Pyramid" structure built on top of CLIP to solve the challenge of Partially Relevant Video Retrieval (PRVR). By organizing event prompts at multiple granularities and enabling structured interactions between them, it outperforms existing models by significant margins across four major benchmarks.
What Problem Does This Paper Address?
Most video retrieval research focuses on Text-to-Video Retrieval (T2VR), where a query is expected to match an entire video. However, real-world scenarios are often more granular: you might be looking for a specific 10-second clip within a 10-minute YouTube video. This is Partially Relevant Video Retrieval (PRVR).
The challenge in PRVR is that a single video can contain dozens of distinct events. Existing methods often process unimodal features (like pre-extracted motion or appearance vectors) in isolation. While Vision-Language Models (VLMs) like CLIP are incredibly powerful, they are designed for images. Simply "plugging in" CLIP to PRVR fails because it doesn't naturally account for the complex temporal relationships between segments—specifically, how small actions (like "pouring a drink") compose larger events (like "having dinner") and how events relate to each other over time.
Methods
The authors propose ProPy (Interactive Prompt Pyramid), which adapts CLIP for PRVR through two main architectural innovations:
- Prompt Pyramid: Instead of a single global token, ProPy uses a set of "event prompts" organized into a pyramid. The bottom layer handles very short segments, while higher layers represent progressively longer segments (using a $2^k$ sampling strategy). This allows the model to capture video semantics at multiple levels of detail simultaneously.
- Ancestor-Descendant (A-D) Interaction Mechanism: To prevent the model from treating segments as independent islands, the authors designed a structured attention mechanism.
- Intra-segment relations: Direct interaction between a "parent" prompt and its "children" (descendants) helps the model understand compositional semantics (e.g., "eating dinner" is made of "sitting down" and "using a fork").
- Inter-segment relations: Interactions between distant segments happen indirectly through higher-level "ancestor" nodes, providing necessary context (e.g., understanding that "drinking again" requires knowing the person drank before) without causing feature leakage or computational bloat.
Additionally, they integrate Temporal Adapters (using 3D-CNNs) to help the frozen CLIP layers understand motion, and use a Multiple Instance Learning (MIL) objective to align the best-matching event prompt with the text query.
Datasets
The model was evaluated on four diverse public datasets:
- TVR: 21.8K videos from TV shows with 5 descriptions each.
- ActivityNet Captions: ~20K YouTube videos focusing on human activities.
- Charades-STA: 6.7K videos with short, fine-grained activity annotations.
- QVHighlights: A challenging dataset for moment retrieval and highlight detection.
Key Insights and Ideas
- The "Easy-to-Hard" Learning Curve: Statistical analysis showed that ProPy naturally learns in stages. It initially focuses on shorter, simpler segments and progressively extends its "understanding" to longer, more complex events as training continues.
- Structure Beats Unstructured Attention: The authors compared their A-D mechanism against "unstructured" attention (where every prompt talks to every other prompt). The structured pyramid performed better, suggesting that hierarchical constraints help the model focus on relevant semantic dependencies rather than getting lost in the noise.
- CLIP Needs Deep, Not Surface, Adaptation: Previous attempts to use CLIP for PRVR often just added a small adapter on top. ProPy shows that "deep" structural changes—integrating prompts directly into the ViT layers—is necessary to fully exploit CLIP’s multi-modal capabilities for video.
Key Results
ProPy achieved state-of-the-art (SOTA) results across the board:
- TVR & ActivityNet: Achieved absolute improvements of 7.7% and 7.2% in R@1 (Recall at rank 1) respectively over previous best models.
- QVHighlights: It reached a 37.4% R@1 score, significantly outperforming competitors even when they used more powerful backbones.
- Efficiency: Despite having more parameters (due to the frozen CLIP backbone), ProPy is highly efficient. Only about 5% of its parameters are trainable, and its inference "matching time" is comparable to much smaller models because the heavy lifting (feature extraction) can be pre-computed.
Limitations and Future Work
The authors highlight a few areas for improvement:
- Memory Footprint: While the trainable parameters are few, the memory required to store or process CLIP features for long videos remains significant.
- Empirical Hyperparameters: The structure of the pyramid (number of layers, segment lengths) is currently set manually. The authors suggest that an adaptive selection method—where the model decides the best pyramid structure for a specific video—would be a promising next step.
- Sampling Strategy: The current $2^k$ sampling is rigid; future versions might benefit from more flexible temporal windowing.
Summary updated