GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement

Zhi-Qi Cheng, Qi Dai, Siyao Li, Teruko Mitamura, Alexander Hauptmann

Abstract

Grounded Situation Recognition (GSR) aims to generate structured semantic summaries of images to achieve human-like event understanding. This task involves detecting a salient activity verb and predicting its corresponding semantic roles and their spatial locations. Traditional methods typically follow a sequential "verb-then-roles" pipeline, which often fails due to the inherent ambiguity of verbs when viewed in isolation. In this paper, we present GSRFormer, a novel two-stage framework that leverages bidirectional relations between verbs and roles. By postponing the final verb decision and employing an alternate semantic attention refinement mechanism, GSRFormer achieves state-of-the-art performance on the SWiG benchmark.

Introduction

Understanding a visual scene requires more than simple object detection; it requires an understanding of the "situation"—the action occurring and the specific roles played by various entities. Grounded Situation Recognition (GSR) formalizes this by requiring models to identify a verb (e.g., cooking), its semantic roles (e.g., agent, tool, container), and the bounding boxes for those roles.

Historically, GSR models have been limited by a rigid, two-stage hierarchical structure: first, classify the verb; second, predict the roles based on that verb. This approach creates a significant bottleneck. If the initial verb classification is incorrect—for instance, mistaking "buying" for "giving"—the subsequent role predictions are fundamentally compromised. Furthermore, this unidirectional flow ignores the fact that roles often provide the necessary context to identify the verb. To address these limitations, we propose GSRFormer, which treats verbs and roles as interdependent components that should be refined simultaneously.

Methodology

GSRFormer introduces a Transformer-based architecture designed to exploit the bidirectional constraints between actions and entities.

1. Transformer-based Encoder

Unlike previous models that commit to a verb early, the GSRFormer encoder learns intermediate representations for a "pseudo-verb" and all potential semantic roles in parallel. We utilize a CNN backbone to extract visual features, which are then fused with learnable verb and role tokens through Multi-Head Attention (MHA). This ensures that the initial features for both the action and the participants are grounded in the global context of the image.

2. Alternate Semantic Attention Refinement

The core of our contribution is the second stage, which employs an iterative refinement process. This stage is built on the principle that semantic roles can disambiguate verbs, and verbs can clarify the nature of roles.

  • Support Verb Retrieval: To provide the model with "common sense" context, we retrieve a set of support images from the training data that share similar semantic features with the target image.
  • Alternate Learning Scheme: We implement a message-passing mechanism where verb features are updated using the nouns detected in the current image, and noun features are updated using the verbs associated with the support set.
  • Iterative Optimization: This refinement process is repeated through five iterative loops. This allows the model to progressively resolve semantic conflicts, such as distinguishing between visually similar activities like "hitting" and "pitching" by analyzing the specific objects and spatial relationships involved.

Experimental Results

We evaluated GSRFormer on the Situations With Grounding (SWiG) benchmark, which contains over 126,000 images annotated with 504 verbs and 190 semantic role types.

Quantitative Performance

GSRFormer established new state-of-the-art (SOTA) benchmarks across all primary metrics:

  • Verb Accuracy: Achieved a Top-1 accuracy of 46.64%, a significant improvement over the previous SOTA (CoFormer) at 44.41%.
  • Value-All Accuracy: In the Top-1 verb setting, role prediction accuracy reached 23.58%.
  • Grounding (grnd-all): The accuracy of correctly identifying roles and their corresponding bounding boxes reached 14.42%, demonstrating superior spatial reasoning and localization.

Key Insights

Our ablation studies revealed that the order of refinement matters: refining nouns first consistently yielded better results than refining verbs first. This suggests that objects are more visually concrete and serve as more reliable anchors for reasoning than abstract action categories. Additionally, the use of support sets proved vital for providing the model with a broader semantic context that individual images might lack.

Conclusion

GSRFormer represents a shift in Grounded Situation Recognition by breaking the "verb-first" bottleneck. By treating verbs and semantic roles as mutually informative, our alternate attention refinement mechanism allows for a more nuanced and accurate understanding of complex visual events. Future work will explore more sophisticated fusion strategies to reduce noise in the refinement loop and extend this bidirectional reasoning framework to video-based event understanding.

Summary updated

Abstract

Grounded Situation Recognition (GSR) aims to generate structured semantic summaries of images for "human-like" event understanding. Specifically, GSR task not only detects the salient activity verb (e.g. buying), but also predicts all corresponding semantic roles (e.g. agent and goods). Inspired by object detection and image captioning tasks, existing methods typically employ a two-stage framework: 1) detect the activity verb, and then 2) predict semantic roles based on the detected verb. Obviously, this illogical framework constitutes a huge obstacle to semantic understanding. First, pre-detecting verbs solely without semantic roles inevitably fails to distinguish many similar daily activities (e.g., offering and giving, buying and selling). Second, predicting semantic roles in a closed auto-regressive manner can hardly exploit the semantic relations among the verb and roles. To this end, in this paper we propose a novel two-stage framework that focuses on utilizing such bidirectional relations within verbs and roles. In the first stage, instead of pre-detecting the verb, we postpone the detection step and assume a pseudo label, where an intermediate representation for each corresponding semantic role is learned from images. In the second stage, we exploit transformer layers to unearth the potential semantic relations within both verbs and semantic roles. With the help of a set of support images, an alternate learning scheme is designed to simultaneously optimize the results: update the verb using nouns corresponding to the image, and update nouns using verbs from support images. Extensive experimental results on challenging SWiG benchmarks show that our renovated framework outperforms other state-of-the-art methods under various metrics.
BibTeX
@inproceedings{cheng2022gsrformer,
  author = {Zhi-Qi Cheng and Qi Dai and Siyao Li and Teruko Mitamura and Alexander Hauptmann},
  title = {{GSRFormer: Grounded Situation Recognition Transformer with Alternate Semantic Attention Refinement}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3547943},
}
Figure