Abstract
This paper strives to classify and detect the relationship between object tubelets appearing within a video as a <subject-predicate-object> triplet. Where existing works treat object proposals or tubelets as single entities and model their relations a posteriori, we propose to classify and detect predicates for pairs of object tubelets a priori. We also propose Social Fabric: an encoding that represents a pair of object tubelets as a composition of interaction primitives. These primitives are learned over all relations, resulting in a compact representation able to localize and classify relations from the pool of co-occurring object tubelets across all timespans in a video. The encoding enables our two-stage network. In the first stage, we train Social Fabric to suggest proposals that are likely interacting. We use the Social Fabric in the second stage to simultaneously fine-tune and predict predicate labels for the tubelets. Experiments demonstrate the benefit of early video relation modeling, our encoding and the two-stage architecture, leading to a new state-of-the-art on two benchmarks. We also show how the encoding enables query-by-primitive-example to search for spatio-temporal video relations.
Introduction
Detecting relationships in videos—represented as <subject-predicate-object> triplets—is significantly more challenging than in static images due to the temporal dimension. Traditional Video Relation Detection (VRD) often relies on a three-stage pipeline: detecting objects in short video snippets, predicting short-term relations, and then using greedy algorithms to "stitch" these into a full-video trajectory. This "bottom-up" approach often fails to capture long-term interactions and struggles with the combinatorial explosion of potential object pairs.
We propose to treat object pairs (tubelets) as the primary unit of analysis from the beginning. Our core innovation is the Social Fabric Encoding (SFE), which represents a pair of object tubelets as a weighted combination of learned "interaction primitives."
Methods
The Social Fabric framework utilizes a two-stage network architecture designed to model interactions across the entire video duration.
1. Social Fabric Encoding (SFE)
Inspired by architectures like NetVLAD, SFE learns a codebook of interaction primitives. A complex relation is encoded by its alignment with these primitives. This allows the model to describe an interaction like "swordfighting" or "chasing" as a sequence of simpler primitives such as "approaching," "running," or "clashing."
2. Two-Stage Network
- Stage 1: Interaction Proposals: The model identifies co-occurring object tubelets that are likely interacting. It employs a 1D watershed algorithm on frame-level interactivity scores to determine the temporal bounds of these interactions.
- Stage 2: Predicate Prediction: Once a proposal is identified, the model uses the SFE to classify the specific predicate (e.g., "kick," "chase," "watch").
The system integrates five types of features: motion (relative trajectories), visual (CNN appearance), language (Word2Vec embeddings), spatio-temporal (I3D), and spatial masks.
Key Insights
The "Social Fabric" approach emphasizes compositionality. By breaking relations down into primitives, the model generalizes better across different scenarios. For example, a "chase" in one video might share motion primitives with a "run" in another.
Furthermore, this encoding enables query-by-primitive-example. Users can provide a few frames of a specific interaction primitive to search for similar complex relations across large video databases.
Results
The model was evaluated on the ImageNet-VidVRD and VidOR (Video Object Relation) benchmarks. Social Fabric set a new state-of-the-art on both:
- VidOR Performance: Achieved an mAP of 11.21 for relation detection, outperforming previous methods like NetRVLAD.
- Long-Term Robustness: The performance advantage of Social Fabric over competitors increases with the duration of the relation, confirming its superior handling of long-range dependencies.
- Feature Importance: Multi-modal integration is crucial; combining all features improved the Relation Tagging (P@1) score from 50.97 (motion only) to 68.86.
Conclusion
Social Fabric demonstrates that a priori modeling of tubelet pairs through interaction primitives significantly improves video relation detection. While the model remains dependent on initial object detector quality, its ability to capture long-term dependencies and represent complex interactions through shared primitives marks a significant advancement in the field.
Summary updated