VisualCOMET: Reasoning About the Dynamic Context of a Still Image

Jae Sung Park, Chandra Bhagavatula, Roozbeh Mottaghi, Ali Farhadi, Yejin Choi

Introduction

Modern computer vision has achieved remarkable success in object recognition and scene classification. However, human perception goes beyond identifying static elements; it is inherently temporal. When observing a still image, humans naturally infer a narrative: what events preceded the moment, what the subjects intend to do, and what the likely consequences will be. VisualCOMET addresses this "cognitive gap" by introducing a framework and a massive dataset designed to push AI toward a more human-like understanding of dynamic visual contexts.

VisualCOMET task overview: reasoning about what happened before, current intents, and what happens next.

The Challenge of Visual Commonsense

While standard image captioning describes the immediate visual state (e.g., "a man struggling in water"), true visual commonsense reasoning requires understanding information that is not explicitly present in the pixels. This "invisible" context involves social interactions, physical cause-and-effect, and temporal flow. The challenge lies in marrying visual perception with broad world knowledge to predict the unobserved past and future.

The VisualCOMET Dataset

To facilitate research in this area, the authors present the VisualCOMET dataset, the first large-scale repository for visual commonsense graphs. The dataset comprises over 1.4 million textual inferences grounded in 59,356 images from the VCR (Visual Commonsense Reasoning) dataset.

Statistics of the VisualCOMET dataset across train, dev, and test splits.

The dataset structure focuses on three temporal dimensions:

  • Before: Actions or states that preceded the image (e.g., "needed to...").
  • Intent: The current motivations of the subjects (e.g., "wanted to...").
  • After: Likely subsequent events (e.g., "will most likely...").

To ensure high-quality annotations, workers used a two-stage process involving video clips of the moments before and after the still frame, grounding the "ground truth" in actual temporal reality rather than speculation.

Methodology

The authors propose a Vision-Language Transformer architecture based on GPT-2 to generate these commonsense inferences. The model utilizes a single-stream architecture, feeding both visual embeddings (global image and person-specific RoI features) and textual embeddings (event and location) into a unified transformer. This allows for tight "co-attention" between visual regions and linguistic tokens.

Key technical innovations include:

  • Person Grounding (PG): Explicitly linking textual descriptions to specific bounding boxes using unique tokens (e.g., [Person1]), which is vital for disambiguating complex social scenes.
  • Event-Place (EP) Loss: A multi-task learning objective where the model predicts the current event and location. This acts as a powerful regularizer, significantly improving the model's ability to reason about the past and future.
  • Nucleus Sampling: A decoding strategy that produces more diverse and less repetitive inferences compared to traditional beam search.

Results and Insights

Experimental results demonstrate that multimodal models (Image + Text) significantly outperform unimodal ones. Visual features provide critical nuances, such as facial expressions or body language, that textual descriptions often omit. The best-performing model achieved a BLEU-2 of 13.50 and a CIDEr score of 18.27. In human evaluations, AI-generated inferences were rated as "likely" 61% of the time, compared to 50% for text-only models.

Despite these advancements, a significant "cognitive gap" remains. There is a ~20-point difference between AI performance and human-generated ground truth. Models still struggle with "hallucinations"—inferring contexts not supported by the image—and the complex interplay of conflicting goals between multiple people.

Conclusion

VisualCOMET represents a significant step toward AI that understands the story behind the pixels. By treating still images as slices of a timeline, the framework enables more sophisticated scene understanding. Future research will likely focus on better integration of external knowledge bases and moving toward multi-step reasoning within visual commonsense graphs.

Summary updated

Abstract

Even from a single frame of a still image, people can reason about the dynamic story of the image before, after, and beyond the frame. For example, given an image of a man struggling to stay afloat in water, we can reason that the man fell into the water sometime in the past, the intent of that man at the moment is to stay alive, and he will need help in the near future or else he will get washed away. We propose VisualCOMET, (Visual Commonsense Reasoning in Time.) the novel framework of visual commonsense reasoning tasks to predict events that might have happened before, events that might happen next, and the intents of the people at present. To support research toward visual commonsense reasoning, we introduce the first large-scale repository of Visual Commonsense Graphs that consists of over 1.4 million textual descriptions of visual commonsense inferences carefully annotated over a diverse set of 59,000 images, each paired with short video summaries of before and after. In addition, we provide person-grounding (i.e., co-reference links) between people appearing in the image and people mentioned in the textual commonsense descriptions, allowing for tighter integration between images and text. We establish strong baseline performances on this task and demonstrate that integration between visual and textual commonsense reasoning is the key and wins over non-integrative alternatives.
BibTeX
@inproceedings{park2020visualcomet,
  author = {Jae Sung Park and Chandra Bhagavatula and Roozbeh Mottaghi and Ali Farhadi and Yejin Choi},
  title = {{VisualCOMET: Reasoning About the Dynamic Context of a Still Image}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  doi = {10.1007/978-3-030-58558-7\_30},
}
Figure