Introduction
Video Relation Detection (VRD) is the task of identifying and localizing object interactions over time, typically represented as triplets like $\langle subject-predicate-object \rangle$. While significant progress has been made in static image relation detection, the temporal dimension of video introduces unique challenges, particularly for interactions that evolve over long durations. Traditional methods often struggle to maintain context across many frames, leading to fragmented or incorrect relationship predictions.
The Challenge of Temporal Context
Most current VRD frameworks rely on a "bottom-up" strategy: dividing videos into short snippets, predicting local relations within those snippets, and then attempting to associate them over time. This approach frequently loses the "big picture." For instance, a complex interaction like a "chase" might be misidentified as simple "running" if the model only sees a one-second window. To overcome this, researchers from the University of Amsterdam proposed the "Social Fabric" framework, which considers the entire trajectory of object pairs—referred to as tubelets—from the outset.
Social Fabric Encoding (SFE)
The centerpiece of this research is the Social Fabric Encoding (SFE). Inspired by dictionary-learning techniques like NetVLAD, SFE represents any interaction between two tubelets as a weighted combination of $K$ learned "interaction primitives." These primitives act as the fundamental building blocks of social interaction, such as "approaching," "touching," or "moving away."
By using these primitives, the model can describe a wide variety of complex relations using a shared, interpretable vocabulary. The system operates in a streamlined two-stage network:
- Interaction Proposals: The model sifts through all possible tubelet pairs to identify those that are actually interacting.
- Predicate Prediction: For the interacting pairs, the model uses the SFE representation—incorporating motion, visual appearance (I3D features), and language embeddings—to classify the specific relationship.
Experimental Results
The "Social Fabric" approach was evaluated on two major benchmarks: ImageNet-VidVRD and the large-scale VidOR dataset. It achieved state-of-the-art performance on both, reaching a mean Average Precision (mAP) of 20.08 on ImageNet-VidVRD and 11.21 on VidOR, significantly outperforming previous snippet-based methods.
A key finding was the model's robustness to interaction length. Unlike snippet-based baselines, which show declining accuracy as interactions grow longer, the Social Fabric method's performance actually improves with duration. This demonstrates its superior ability to capture the full temporal evolution of social interactions that unfold over many seconds.
Conclusion and Future Work
By treating video relations as a "fabric" woven from basic primitives, this work provides a powerful new lens for understanding dynamic scenes. While currently reliant on external object detectors and trackers, future iterations could integrate tubelet generation directly into the pipeline to handle noisy detections more robustly. Furthermore, the interpretable nature of the learned primitives enables advanced "query-by-example" video retrieval, where users can search for specific interactions across massive databases using simple visual examples.
Summary updated