Abstract
Visual Relationship Forecasting (VRF) aims to anticipate relations among objects without observing future visual content. The task relies on capturing and modeling the semantic coherence in object interactions, as it underpins the evolution of events and scenes in videos. However, existing VRF datasets offer limited support for learning such coherence due to noisy annotations and weak correlations between actions and relationship transitions. Furthermore, existing methods struggle to distinguish similar relationships and overfit to unchanging relationships in consecutive frames. To address these challenges, we present SemCoBench, a benchmark that emphasizes semantic coherence for visual relationship forecasting. Based on action labels and short-term subject-object pairs, SemCoBench decomposes relationship categories and dynamics by cleaning and reorganizing video datasets. In addition, we present the Semantic Coherent Transformer (SemCoFormer) to model semantic coherence with a Relationship Augmented Module (RAM) and a Coherence Reasoning Module (CRM). Experimental results on SemCoBench demonstrate that modeling semantic coherence is a key step toward reasonable, fine-grained, and diverse visual relationship forecasting.
Introduction: The Hierarchy of Video Understanding
Visual Relationship Forecasting (VRF) is a sophisticated task in computer vision that involves predicting future interactions between objects—represented as <subject-predicate-object> triplets—based solely on past observations. Unlike simple action recognition, VRF requires a deep understanding of the scene's "logic" or semantic coherence.
As shown in the hierarchy above, VRF (Relation-Level Forecasting) sits between high-level action forecasting and low-level object localization. It provides a structured, interpretable way to understand how a scene evolves. For instance, seeing a person reach for a refrigerator should lead a model to forecast the transition from "not contacting" to "opening" or "touching."
The Roadblocks: Noise and Semantic Blindness
The authors identify two primary obstacles hindering progress in VRF:
- Noisy and Static Datasets: Many existing benchmarks are dominated by "static" relationships (e.g., "person-in-front-of-wall") that persist across hundreds of frames. This encourages models to simply predict the current state for the future, failing to capture actual interaction dynamics.
- Semantic Ambiguity: Visually, "holding" an object can look nearly identical to "carrying" or "touching" it. Without semantic context—knowing the identity of the object and the likely flow of the action—models struggle to choose the correct fine-grained predicate.
SemCoBench: A Refined Foundation
To solve the data quality issue, the researchers introduced SemCoBench. This benchmark is derived from cleaning and reorganizing the Action Genome (AG) and Video Visual Relation Detection (VidOR) datasets.
As illustrated in Figure 2, SemCoBench filters out noisy spatial predicates and focuses on high-quality relationship categories (13 for SemCo-AG and 35 for SemCo-VidOR). By reorganizing the data around action labels and subject-object pairs, the benchmark ensures that models must actually perform semantic reasoning to succeed, rather than relying on temporal persistence.
The SemCoFormer Architecture
The proposed Semantic Coherent Transformer (SemCoFormer) introduces two key modules to tackle semantic blindness and temporal overfitting:
Relationship Augmented Module (RAM)
The RAM facilitates a "handshake" between visual features and semantic embeddings (e.g., word2vec). By integrating the meaning of an object category (like "refrigerator") with its visual appearance, the model gains a prior on which interactions are logically possible. This helps distinguish between visually similar but semantically distinct actions.
Coherence Reasoning Module (CRM)
To combat the "lazy model" trap where a network simply predicts no change, the CRM employs sparse coding. Instead of processing every consecutive frame, it samples non-consecutive temporal blocks. This forces the Transformer to learn the underlying dynamics and transitions of an event rather than overfitting to the visual similarity of adjacent frames.
Key Results and Insights
SemCoFormer achieves state-of-the-art (SOTA) performance across the new benchmarks:
- SemCo-VidOR: Reached 36.84% Accuracy and 33.59% mAP, significantly outperforming the previous SOTA, SceneSayerSDE.
- SemCo-AG: Achieved 75.50% Accuracy and 80.94% mAP.
Qualitative analysis shows that SemCoFormer is particularly adept at predicting specific transitions, such as an "adult-caress-cat," where baseline models often default to generic labels like "hug."
Conclusion
The SemCo framework demonstrates that "semantic coherence"—the logical flow of interactions—is the missing link in robust visual relationship forecasting. By refining datasets to focus on meaningful transitions and using semantic-aware Transformer modules, AI can move beyond simple pattern matching toward a more human-like understanding of how the world unfolds.
Summary updated