CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling

Abhidip Bhattacharyya, Martha Palmer, Christoffer HeckmanACL 2023

TL;DR

This paper introduces CRAPES, a novel framework for Grounded Situation Recognition (GSR) that breaks free from the limitation of predicting only a single action per image. By leveraging cross-modal annotation projection and off-the-shelf vision-language and semantic role labeling components, CRAPES significantly improves performance on the SWiG dataset and demonstrates the potential for more comprehensive image understanding.

What Problem Does This Paper Address?

Current approaches to Grounded Situation Recognition (GSR) primarily focus on identifying a single action and its participants within an image. This is largely due to the constraints of existing datasets like SWiG, which are annotated with only one verb frame per image. However, real-world scenes often contain multiple, overlapping actions, and limiting GSR to a single action misses a significant amount of semantic information. This paper addresses this limitation by proposing a more flexible formulation that allows for the detection of multiple co-existing verb frames within a single image, drawing inspiration from the richer semantic representations found in text-based Semantic Role Labeling (SRL).

Methods

CRAPES (Cross-modal Annotation Projection for Visual Semantic Role Labeling) is built upon two core modules:

  1. Vision-Language (V+L) Model: The authors utilize Oscar, a transformer-based architecture pre-trained on large image-text datasets (Flickr30k and SWiG). Oscar generates image-text representations and provides attention weights that are crucial for grounding.
  2. Text-based Semantic Role Labeling (SRL) System: Two types of SRL systems are employed: an off-the-shelf FrameNet parser (LOME) and a custom BERT-based SRL model trained on SWiG frames. These systems process text generated from the V+L model to identify verbs and their associated semantic roles.

The key innovation lies in Cross-modal Annotation Projection (AP). This technique, inspired by cross-lingual SRL, uses the attention weights from the V+L model to project the semantic role annotations from the text domain back to the image, specifically to bounding boxes. This allows CRAPES to align predicted roles with visual entities in the image. The framework is designed to be modular, allowing for the reuse of general-purpose V+L and SRL components, which aids in generalization and reduces the need for extensive manual annotation.

Datasets

  • SWiG: Used for both training and evaluation. This dataset provides images with ground truth annotations for semantic roles and bounding boxes, but is limited to one verb frame per image.
  • Flickr30k: Used for pre-training the Oscar V+L model. This dataset contains images with textual descriptions but does not provide explicit semantic role annotations for images.

Key Insights and Ideas

  • Breaking the Single-Action Constraint: The most significant insight is that by relaxing the single-verb-frame-per-image constraint, image comprehension can be dramatically improved. This aligns better with the complexity of real-world scenes.
  • Leveraging Off-the-Shelf Components: The success of CRAPES highlights the power of using pre-trained, general-purpose V+L and SRL models. This approach not only improves performance but also enhances generalization capabilities to out-of-domain frames.
  • Cross-modal Annotation Projection (AP) for Grounding: The use of attention weights from the V+L model as a mechanism to project semantic role information from text to image bounding boxes is a novel and effective technique for grounding.
  • Modularity for Flexibility: The independent nature of the V+L and SRL modules allows CRAPES to be adapted to different SRL paradigms (like FrameNet or PropBank) and to be trained on diverse datasets, increasing its versatility.

Key Results

CRAPES achieved a substantial 28.6 point jump in performance on the SWiG dataset compared to previous state-of-the-art methods, representing a 76% relative gain in the "value" metric. It also showed significant improvements in "value-all" (31% relative gain) and "grnd" (15% relative gain). The paper presents two versions: CRAPES1, which can accommodate co-existing verb frames, and CRAPES2, which adheres to the single-frame-per-image constraint of SWiG. CRAPES1 demonstrated the ability to detect out-of-domain frames and parallel frames not present in the ground truth.

Limitations and Future Work

  • "grnd-all" Metric: The paper acknowledges that CRAPES lags in the "grnd-all" metric, which requires accurate grounding for all predicted nouns within a role set. This is attributed to potential distribution shifts between the V+L and SRL modules and the interpretability of attention weights.
  • Attention Layer Investigation: Further investigation is needed into which attention layers are most effective for grounding.
  • Extending to Other Datasets: The authors plan to extend their experiments to datasets like Flickr30k, which present new challenges due to the lack of explicit semantic role annotations for images and the presence of more abstract conceptual roles.
  • Developing New Metrics: For multi-frame scenarios, current evaluation metrics are insufficient. The development of more robust and appropriate metrics is a critical future direction.
  • Human Evaluation: A more rigorous study with human evaluation is suggested to fully assess the capabilities of CRAPES, especially for multi-frame scenarios.

Summary updated

BibTeX
@inproceedings{bhattacharyya2023crapes,
  author = {Abhidip Bhattacharyya and Martha Palmer and Christoffer Heckman},
  title = {{CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure