Introduction
Visual Relationship Detection (VRD) is a fundamental computer vision task that involves identifying triplets in the form of $\langle subject, predicate, object \rangle$. Traditional VRD models typically operate within a "closed-set" paradigm, where they are limited to recognizing categories present in the training data. However, real-world applications demand Open-Vocabulary capabilities—the ability to detect "novel" relationships (e.g., a "hat" being "worn" by a "person") that were not encountered during the training phase.
The current state-of-the-art for open-vocabulary tasks often leverages CLIP (Contrastive Language-Image Pre-training), a model trained on vast quantities of image-text pairs. Despite its success in static image tasks, CLIP faces significant challenges when applied to video-based relationship detection. First, there is a substantial domain gap: CLIP is optimized for static images and global descriptions, making it less effective at capturing fine-grained object-level interactions and the temporal dynamics inherent in video. Second, the problem of semantic discovery arises; using CLIP’s fixed vocabulary makes it difficult to align novel video-specific predicates that require integrated spatial and temporal context.
Methodology
To address these challenges, researchers have proposed a multi-modal prompting framework designed to adapt CLIP for Open-Vocabulary Video Visual Relationship Detection (Open-VidVRD) without the need for computationally expensive full-model fine-tuning.
Spatio-temporal Visual Prompting
To enable CLIP to process relational dynamics, the model extracts features from four distinct regions: the subject, the object, their union (the bounding box encompassing both), and the background.
- Spatial Modeling: Transformer blocks are employed to model interactions between these four regions within individual frames. "Role embeddings" are introduced to distinguish between the subject and the object.
- Temporal Modeling: These spatial features are then processed across time using a secondary set of Transformers. This captures how relationships evolve over the duration of a video segment, distinguishing between actions like "walking toward" and "walking away."
Vision-guided Language Prompting
Rather than relying on static text prompts (e.g., "a photo of [predicate]"), the framework introduces a dynamic language prompting strategy:
- Learnable Continuous Prompts: Static tokens that capture general task-specific context.
- Learnable Conditional Prompts: These prompts are generated dynamically based on the visual features of the video.
By interlacing these prompts, the language encoder becomes "aware" of the specific visual context, significantly improving the identification of novel predicates.
Experimental Evaluation
The proposed method was evaluated on two benchmark datasets: VidVRD (1,000 videos, 35 object categories, 132 predicate categories) and VidOR (10,000 videos, 80 object categories, 50 predicate categories). In both instances, datasets were split into "base" categories for training and "novel" categories for zero-shot testing.
Results and Discussion
The proposed framework significantly outperformed existing baselines, particularly in the detection of novel relationships. On the VidVRD dataset, the model achieved a mean Average Precision (mAP) of 15.99% for novel categories, a substantial improvement over the previous state-of-the-art (RePro) at 6.10%. When utilizing ground-truth object trajectories, the mAP for novel categories reached 21.14%, nearly doubling prior performance.
Qualitative analysis via t-SNE visualizations demonstrates that the prompting strategy effectively organizes CLIP’s latent space for the video domain. As shown in the comparison above, features for novel predicates such as "swim behind" or "stand next to" are much more tightly clustered and distinct compared to standard CLIP features. This confirms that the multi-modal prompts successfully bridge the gap between static image knowledge and dynamic video relationships.
Conclusion
This research demonstrates that multi-modal prompting is a powerful tool for adapting large-scale vision-language models to complex, open-vocabulary video tasks. By decoupling spatial and temporal modeling and introducing vision-guided language prompts, the model achieves significant performance gains in detecting unseen relationships. A current limitation is the reliance on an external tracklet detector; future work will focus on developing an end-to-end pipeline to integrate tracking and relationship detection more seamlessly.
Summary updated