Abstract: Real-world scenarios often require the anticipation of object interactions in unknown future, which would assist the decision-making process of both humans and agents. To meet this challenge, we present a new task named Visual Relationship Forecasting (VRF) in videos to explore the prediction of visual relationships in a reasoning manner. Specifically, given a subject-object pair with H existing frames, VRF aims to predict their future interactions for the next T frames without visual evidence. To evaluate the VRF task, we introduce two video datasets named VRF-AG and VRF-VidOR, with a series of spatio-temporally localized visual relation annotations in a video. These two datasets densely annotate 13 and 35 visual relationships in 1923 and 13447 video clips, respectively. In addition, we present a novel Graph Convolutional Transformer (GCT) framework, which captures both object-level and frame-level dependencies by spatio-temporal Graph Convolution Network and Transformer. Experimental results on both VRF-AG and VRF-VidOR datasets demonstrate that GCT outperforms the state-of-the-art sequence modelling methods on visual relationship forecasting.
Introduction
Predicting the future in video is a cornerstone of computer vision, but most research focuses on "what" (action recognition) or "where" (trajectory prediction). Visual Relationship Forecasting (VRF) tackles the "how" — the fine-grained interactions between subjects and objects, represented as <subject-predicate-object> triplets. This task is essential for autonomous agents that need to anticipate human intent and interact safely in dynamic environments.
As shown in Figure 1, VRF sits at the center of a hierarchy, bridging the gap between high-level action forecasting and low-level object localization. By predicting the evolution of relationships, models can better understand the semantic flow of a scene.
The Challenges of VRF
Despite its importance, VRF research faces three major hurdles:
- Noisy Data: Existing datasets often contain "lazy" annotations or spatial relationships (like "in front of") that remain static, providing little signal for learning actual dynamics.
- The "Static Frame" Trap: Most models overfit to consecutive frames where nothing changes. If a person is holding a box in the first few frames, a naive model will likely predict "holding" indefinitely, missing the transition when they put it down.
- Visual Ambiguity: Visual cues alone are often insufficient. Distinguishing between "holding" and "carrying" requires understanding the semantic context of the objects involved.
Figure 2 illustrates these challenges. In traditional benchmarks like Action Genome, relationships are often redundant and static. The proposed VRFBench (SemCoBench) focuses on capturing meaningful transitions, such as moving from "not contacting" to "holding."
The SemCo Framework
To address these challenges, we introduce the Semantic Coherent Transformer (SemCoFormer). This architecture is designed to prioritize semantic logic and temporal transitions over raw pixel replication. It consists of three main components:
- Feature Representation Module (FRM): Extracts a multi-modal feature set including visual features (via Inception-ResNet-V2), spatial features (bounding box deltas), and semantic features (word2vec embeddings of object labels).
- Relationship Augmented Module (RAM): This bidirectional attention mechanism uses Visual-Augment-Semantic (VAS) to update object meanings based on the scene and Semantic-Augment-Visual (SAV) to focus the visual model on regions relevant to specific objects. For instance, knowing a "refrigerator" is present helps the model narrow down likely predicates like "opening."
- Coherence Reasoning Module (CRM): To solve the "Static Frame" trap, we employ Sparse Coding. Instead of processing every consecutive frame, the model groups non-consecutive frames, forcing the Transformer to attend to changes and transitions across the sequence.
SemCoBench: A Refined Benchmark
We introduced SemCoBench, a benchmark specifically designed for semantic coherence. It is derived from two major sources: SemCo-AG (a cleaned version of Action Genome with 13 predicates) and SemCo-VidOR (based on the Video Visual Relation Detection dataset with 35 predicates). These datasets were manually filtered to ensure that relationships are closely related and predictable, removing the noise that plagues earlier VRF benchmarks.
Key Results and Insights
SemCoFormer sets a new state-of-the-art (SOTA) on both SemCoBench datasets. On SemCo-VidOR, it achieved 36.84% Accuracy and 33.59% mAP, significantly outperforming previous sequence modeling methods. On SemCo-AG, it reached 75.50% Accuracy and 80.94% mAP.
Qualitative analysis shows that while baseline models often get "stuck" in a single state, SemCoFormer correctly anticipates transitions like "speaking to" or "caressing." This success stems from two key ideas:
- Semantics as a Prior: Object labels act as a powerful constraint, providing a "common sense" filter for predictions.
- The Power of Sparsity: By downsampling the temporal dimension, the model highlights the deltas in the video, preventing it from simply replicating the last observed state.
Conclusion and Future Work
VRF serves as a necessary bridge between raw pixels and complex activity understanding. While SemCoFormer shows significant promise, challenges remain in handling the multi-modality of the future—where a person could take multiple valid actions. Future work will explore generative models and longer-term temporal dependencies to further improve the reasonableness and diversity of predicted interactions.
Summary updated