Cognitive Visual Commonsense Reasoning Using Dynamic Working Memory

Xuejiao Tang, Xin Huang, Wenbin Zhang, Travers B. Child, Qiong Hu, Zhen Liu, Ji Zhang

TL;DR

This paper introduces DMVCR (Dynamic working Memory based cognitive Visual Commonsense Reasoning), a novel framework that enhances a model's ability to explain why a visual scene is happening. By incorporating a "dictionary unit" that acts as a dynamic knowledge base, the model stores and references accumulated commonsense to outperform previous state-of-the-art methods on the challenging VCR benchmark.

What Problem Does This Paper Address?

While standard Visual Question Answering (VQA) asks "What is in this image?", Visual Commonsense Reasoning (VCR) asks "Why is this happening?" and "What will happen next?". This requires more than just object detection; it requires "cognition-level" understanding—the ability to infer intentions, emotions, and social contexts.

The authors identify a major gap in current VCR approaches:

  1. Sequence-based models (like LSTMs) struggle to encode long-term dependencies and lack external "prior knowledge."
  2. Pre-trained models (like ViLBERT) are powerful but often lack generalizability across different types of reasoning.

Humans use "working memory" to store experiences and apply them to new situations. This paper seeks to mimic that process by giving AI a way to store and look up "commonsense" during inference.

Methods

The authors propose the DMVCR network, which consists of four distinct layers:

  1. Feature Representation Layer: Uses BERT to embed text (questions and answers) and ResNet50 to extract features from detected objects in the image. It uses "visual grounding" tags (e.g., [person1]) to link specific words to image regions.
  2. Multimodal Fusion Layer: This layer blends text and image data. It uses a BiLSTM to capture sequence context and an Attention Module to ensure the model doesn't ignore subtle object features that might be crucial for the rationale.
  3. Encoder Layer (The Core Innovation): Beyond a standard RNN, this layer includes a Dictionary Unit. This unit is a $d \times k$ matrix that acts as a dynamic knowledge base. During training, the model learns to store "commonsense" representations in this dictionary. During inference, it performs a "look-up" to retrieve relevant prior knowledge to help it reason.
  4. Prediction Layer: A multi-layer perceptron (MLP) that outputs a probability distribution over the four multiple-choice options.

Datasets

The model was evaluated on the VCR Dataset, a massive benchmark containing:

  • 290,000 multiple-choice questions.
  • 290,000 correct answers and rationales.
  • 110,000 unique images derived from movie clips.
  • Inference Types: The dataset covers various reasoning types, including Explanation (38%), Activity (24%), and others like Mental State, Temporal, and Scene context.

Key Insights and Ideas

  • Dictionary as "Experience": The most significant idea is treating the dictionary unit as a form of "prior knowledge." Instead of relying solely on the current image/question pair, the model learns a global set of "commonsense features" that it can reference.
  • Rationale over Answers: The authors found that their model provided a much larger performance boost in predicting the rationale ($QA \to R$) than just the answer ($Q \to A$). This suggests that the dynamic memory is particularly helpful for the "why" part of the task.
  • Mimicking Human Cognition: By using a "look-up" and "update" mechanism for the memory cell, the architecture attempts to simulate how humans recall similar past scenarios to interpret a current one.

Key Results

The DMVCR model was compared against several popular baselines, including RevisitedVQA, BottomUpTopDown, and the previous baseline SOTA, R2C.

Subtask DMVCR Accuracy Improvement over R2C
$Q \to A$ (Answer) 62.4% +0.8%
$QA \to R$ (Rationale) 67.5% +7.5%
$Q \to AR$ (Both) 42.3% +8.2%

The significant jump in the $Q \to AR$ score (which requires getting both the answer and the rationale right) demonstrates that the dictionary unit effectively bridges the gap between seeing an object and understanding its purpose.

Limitations and Future Work

While the model shows strong quantitative gains, the qualitative analysis reveals some "hallucination" or logic gaps. For example, in some cases, the model correctly identifies an action (e.g., "they are preparing to run from a fire") but selects a flawed rationale (e.g., "they are turned towards the fire").

Future directions suggested by the paper include: * Investigating more sophisticated multimodal fusion methods. * Refining the attention mechanism to better capture the relationship between specific objects and the "commonsense" stored in the dictionary. * Exploring how to make the "dictionary" more interpretable so researchers can see exactly what "commonsense" the model is retrieving.

Summary updated

Abstract

Visual Commonsense Reasoning (VCR) predicts an answer with corresponding rationale, given a question-image input. VCR is a recently introduced visual scene understanding task with a wide range of applications, including visual question answering, automated vehicle systems, and clinical decision support. Previous approaches to solving the VCR task generally rely on pre-training or exploiting memory with long dependency relationship encoded models. However, these approaches suffer from a lack of generalizability and prior knowledge. In this paper we propose a dynamic working memory based cognitive VCR network, which stores accumulated commonsense between sentences to provide prior knowledge for inference. Extensive experiments show that the proposed model yields significant improvements over existing methods on the benchmark VCR dataset. Moreover, the proposed model provides intuitive interpretation into visual commonsense reasoning. A Python implementation of our mechanism is publicly available at https://github.com/tanjatang/DMVCR
BibTeX
@inproceedings{tang2021cognitive,
  author = {Xuejiao Tang and Xin Huang and Wenbin Zhang and Travers B. Child and Qiong Hu and Zhen Liu and Ji Zhang},
  title = {{Cognitive Visual Commonsense Reasoning Using Dynamic Working Memory}},
  booktitle = {DaWaK},
  year = {2021},
  url = {https://openreview.net/forum?id=p5i1lxWdySZ},
}

Loading related content...

Figure