Cognitive Visual Commonsense Reasoning Using Dynamic Working Memory

Xuejiao Tang, Xin Huang, Wenbin Zhang, Travers B. Child, Qiong Hu, Zhen Liu, Ji Zhang

Abstract

Visual Commonsense Reasoning (VCR) predicts an answer with corresponding rationale, given a question-image input. VCR is a recently introduced visual scene understanding task with a wide range of applications, including visual question answering, automated vehicle systems, and clinical decision support. Previous approaches to solving the VCR task generally rely on pre-training or exploiting memory with long dependency relationship encoded models. However, these approaches suffer from a lack of generalizability and prior knowledge. In this paper we propose a dynamic working memory based cognitive VCR network, which stores accumulated commonsense between sentences to provide prior knowledge for inference. Extensive experiments show that the proposed model yields significant improvements over existing methods on the benchmark VCR dataset. Moreover, the proposed model provides intuitive interpretation into visual commonsense reasoning.

Introduction

Visual Commonsense Reasoning (VCR) represents a significant step forward from standard Visual Question Answering (VQA). While VQA typically focuses on identifying objects or attributes (e.g., "What is in this image?"), VCR requires "cognition-level" understanding to answer "Why is this happening?" and "What will happen next?". This task demands the ability to infer intentions, emotions, and social contexts from visual cues.

Current approaches to VCR face two primary challenges. First, sequence-based models like LSTMs often struggle to encode long-term dependencies and lack access to external prior knowledge. Second, while large-scale pre-trained models are powerful, they can lack generalizability across diverse reasoning types. To address these issues, we introduce DMVCR (Dynamic working Memory based cognitive Visual Commonsense Reasoning), a framework that mimics human "working memory" by storing and referencing accumulated commonsense during inference.

The DMVCR Framework

The DMVCR network is structured into four distinct layers designed to integrate visual and textual information while leveraging a dynamic knowledge base.

  1. Feature Representation Layer: This layer utilizes BERT for text embedding (questions and answers) and ResNet50 for extracting features from detected objects. Visual grounding tags (e.g., [person1]) are used to explicitly link textual references to specific image regions.
  2. Multimodal Fusion Layer: This layer blends textual and visual data using a BiLSTM to capture sequence context. An Attention Module is employed to ensure the model focuses on subtle object features that are critical for generating rationales.
  3. Encoder Layer: The core innovation of DMVCR is the inclusion of a Dictionary Unit within the encoder. This unit acts as a dynamic knowledge base. During training, the model learns to store commonsense representations in this dictionary. During inference, it performs a "look-up" to retrieve relevant prior knowledge, which is then updated to reflect new information.
  4. Prediction Layer: A multi-layer MLP outputs the final probability distribution over the multiple-choice options.
The DMVCR architecture featuring the dynamic dictionary unit.

Dataset and Evaluation

The model was evaluated on the VCR dataset, a comprehensive benchmark consisting of 290,000 multiple-choice questions and rationales derived from 110,000 unique movie clip images. The dataset is diverse, covering various reasoning categories such as Explanation, Activity, Mental State, Temporal, and Scene context.

Distribution of reasoning types in the VCR dataset.

Results and Discussion

DMVCR was compared against several baselines, including RevisitedVQA, BottomUpTopDown, and the previous state-of-the-art, R2C. The results demonstrate that DMVCR significantly outperforms these methods, particularly in tasks requiring rationale generation.

Subtask DMVCR Accuracy Improvement over R2C
$Q \to A$ (Answer) 62.4% +0.8%
$QA \to R$ (Rationale) 67.5% +7.5%
$Q \to AR$ (Both) 42.3% +8.2%

The substantial improvement in the $QA \to R$ and $Q \to AR$ metrics suggests that the dynamic memory unit is highly effective at bridging the gap between visual perception and logical reasoning. By treating the dictionary as a repository of "experience," the model can reference a global set of commonsense features rather than relying solely on the immediate input pair.

Conclusion

DMVCR introduces a novel way to incorporate prior knowledge into visual reasoning tasks through a dynamic working memory. While the model shows strong quantitative gains, qualitative analysis indicates room for improvement in resolving logic gaps where the model identifies an action correctly but provides a flawed rationale. Future work will focus on more sophisticated multimodal fusion, refining attention mechanisms, and enhancing the interpretability of the dictionary unit.

Summary updated

Abstract

Visual Commonsense Reasoning (VCR) predicts an answer with corresponding rationale, given a question-image input. VCR is a recently introduced visual scene understanding task with a wide range of applications, including visual question answering, automated vehicle systems, and clinical decision support. Previous approaches to solving the VCR task generally rely on pre-training or exploiting memory with long dependency relationship encoded models. However, these approaches suffer from a lack of generalizability and prior knowledge. In this paper we propose a dynamic working memory based cognitive VCR network, which stores accumulated commonsense between sentences to provide prior knowledge for inference. Extensive experiments show that the proposed model yields significant improvements over existing methods on the benchmark VCR dataset. Moreover, the proposed model provides intuitive interpretation into visual commonsense reasoning. A Python implementation of our mechanism is publicly available at https://github.com/tanjatang/DMVCR
BibTeX
@inproceedings{tang2021cognitive,
  author = {Xuejiao Tang and Xin Huang and Wenbin Zhang and Travers B. Child and Qiong Hu and Zhen Liu and Ji Zhang},
  title = {{Cognitive Visual Commonsense Reasoning Using Dynamic Working Memory}},
  booktitle = {DaWaK},
  year = {2021},
  url = {https://openreview.net/forum?id=p5i1lxWdySZ},
}
Figure