Abstract
Grounded Situation Recognition (GSR) aims to generate structured semantic summaries of images to achieve human-like event understanding. This task involves detecting a salient activity verb and predicting its corresponding semantic roles and their spatial locations. Traditional methods typically follow a sequential "verb-then-roles" pipeline, which often fails due to the inherent ambiguity of verbs when viewed in isolation. In this paper, we present GSRFormer, a novel two-stage framework that leverages bidirectional relations between verbs and roles. By postponing the final verb decision and employing an alternate semantic attention refinement mechanism, GSRFormer achieves state-of-the-art performance on the SWiG benchmark.
Introduction
Understanding a visual scene requires more than simple object detection; it requires an understanding of the "situation"—the action occurring and the specific roles played by various entities. Grounded Situation Recognition (GSR) formalizes this by requiring models to identify a verb (e.g., cooking), its semantic roles (e.g., agent, tool, container), and the bounding boxes for those roles.
Historically, GSR models have been limited by a rigid, two-stage hierarchical structure: first, classify the verb; second, predict the roles based on that verb. This approach creates a significant bottleneck. If the initial verb classification is incorrect—for instance, mistaking "buying" for "giving"—the subsequent role predictions are fundamentally compromised. Furthermore, this unidirectional flow ignores the fact that roles often provide the necessary context to identify the verb. To address these limitations, we propose GSRFormer, which treats verbs and roles as interdependent components that should be refined simultaneously.
Methodology
GSRFormer introduces a Transformer-based architecture designed to exploit the bidirectional constraints between actions and entities.
1. Transformer-based Encoder
Unlike previous models that commit to a verb early, the GSRFormer encoder learns intermediate representations for a "pseudo-verb" and all potential semantic roles in parallel. We utilize a CNN backbone to extract visual features, which are then fused with learnable verb and role tokens through Multi-Head Attention (MHA). This ensures that the initial features for both the action and the participants are grounded in the global context of the image.
2. Alternate Semantic Attention Refinement
The core of our contribution is the second stage, which employs an iterative refinement process. This stage is built on the principle that semantic roles can disambiguate verbs, and verbs can clarify the nature of roles.
- Support Verb Retrieval: To provide the model with "common sense" context, we retrieve a set of support images from the training data that share similar semantic features with the target image.
- Alternate Learning Scheme: We implement a message-passing mechanism where verb features are updated using the nouns detected in the current image, and noun features are updated using the verbs associated with the support set.
- Iterative Optimization: This refinement process is repeated through five iterative loops. This allows the model to progressively resolve semantic conflicts, such as distinguishing between visually similar activities like "hitting" and "pitching" by analyzing the specific objects and spatial relationships involved.
Experimental Results
We evaluated GSRFormer on the Situations With Grounding (SWiG) benchmark, which contains over 126,000 images annotated with 504 verbs and 190 semantic role types.
Quantitative Performance
GSRFormer established new state-of-the-art (SOTA) benchmarks across all primary metrics:
- Verb Accuracy: Achieved a Top-1 accuracy of 46.64%, a significant improvement over the previous SOTA (CoFormer) at 44.41%.
- Value-All Accuracy: In the Top-1 verb setting, role prediction accuracy reached 23.58%.
- Grounding (grnd-all): The accuracy of correctly identifying roles and their corresponding bounding boxes reached 14.42%, demonstrating superior spatial reasoning and localization.
Key Insights
Our ablation studies revealed that the order of refinement matters: refining nouns first consistently yielded better results than refining verbs first. This suggests that objects are more visually concrete and serve as more reliable anchors for reasoning than abstract action categories. Additionally, the use of support sets proved vital for providing the model with a broader semantic context that individual images might lack.
Conclusion
GSRFormer represents a shift in Grounded Situation Recognition by breaking the "verb-first" bottleneck. By treating verbs and semantic roles as mutually informative, our alternate attention refinement mechanism allows for a more nuanced and accurate understanding of complex visual events. Future work will explore more sophisticated fusion strategies to reduce noise in the refinement loop and extend this bidirectional reasoning framework to video-based event understanding.
Summary updated