Abstract
A vexing problem in artificial intelligence is reasoning about events that occur in complex, changing visual stimuli, such as in video analysis or gameplay. Inspired by a rich tradition of visual reasoning and memory in cognitive psychology and neuroscience, we developed an artificial, configurable visual question and answer dataset (COG) to parallel experiments in humans and animals. COG is much simpler than the general problem of video analysis, yet it addresses many of the problems relating to visual and logical reasoning and memory—problems that remain challenging for modern deep learning architectures. We additionally propose a deep learning architecture that performs competitively on other diagnostic VQA datasets (i.e., CLEVR) as well as easy settings of the COG dataset. However, several settings of COG result in datasets that are progressively more challenging to learn. After training, the network can zero-shot generalize to many new tasks. Preliminary analyses of the network architectures trained on COG demonstrate that the network accomplishes the task in a manner interpretable to humans.
The Challenge of Temporal Visual Reasoning
While Deep Learning has made massive strides in Visual Question Answering (VQA), most benchmarks focus on static images. In these settings, a model looks at one picture and answers a logic-based question. However, real-world intelligence—like navigating a city or playing a video game—requires reasoning across time.
Researchers from NYU and Google Brain identify two major gaps in current VQA research:
- The "Shortcut" Problem: Models often exploit statistical biases in natural datasets to "cheat" rather than actually reasoning.
- The Memory Gap: Most models lack a functional "working memory" to store relevant visual details from the past, ignore irrelevant distractors, and update their internal state as new information arrives.
Introducing the COG Dataset
To address these gaps, the authors developed COG (COnfigurable visual question and answer). COG consists of sequences of synthetic images (frames) and instructions that require memory and temporal logic. Unlike static datasets, COG forces models to track objects across time and handle conditional instructions.
Key Features of COG: * Scale: It features 44 tasks and over 2 trillion possible task instances. * Configurability: Researchers can tune difficulty by changing the number of frames, the memory duration required, and the number of "distractor" objects. * Compositionality: Tasks are built from 8 basic operators (Select, GetColor, Exist, etc.), allowing for rigorous testing of logical combinations.
A Neuro-Inspired Architecture with vSTM
The proposed multi-modal recurrent architecture is inspired by cognitive neuroscience, consisting of four main components:
- Visual System: A CNN that processes images, utilizing FiLM (Feature-wise Linear Modulation) and spatial attention to focus on specific attributes like color or shape.
- Semantic System: A bidirectional LSTM that encodes text instructions into a "semantic memory."
- The Controller: A Gated Recurrent Unit (GRU) that acts as the "brain," integrating information from all systems and generating attention signals.
- Visual Short-Term Memory (vSTM): This is the core innovation. It is a convolutional LSTM variant where the gating (input, forget, and output) is controlled externally by the GRU. This allows the model to explicitly decide what to remember from previous frames and what to discard.
Crucially, the model is allowed multiple "pondering" steps per image, letting it perform iterative reasoning before producing an output.
Key Insights and Generalization
One of the most impressive findings is the model's ability for zero-shot generalization. Because the model learns fundamental building blocks (operators), it can solve entirely new tasks it wasn't trained on by recombining learned operations. For example, if trained on "finding the last red object" and "identifying the color of a square," it can generalize to "identifying the color of the last red square."
Furthermore, the model's reasoning is interpretable. By visualizing attention maps during "pondering" steps, researchers observed the model decomposing complex questions—first identifying a target class of objects and then shifting attention to specific spatial or temporal properties.
Results and Performance
- CLEVR Performance: The model achieved 96.8% accuracy, surpassing human-level performance (92.6%) and rivaling specialized state-of-the-art architectures.
- COG Performance: In the canonical setting, the model reached 93.7% accuracy.
- Zero-Shot Success: When trained on 43 tasks and tested on a 44th unseen task, the model averaged 85.4% accuracy, far above the chance level of 26.7%.
- Stress Testing: While accuracy remains high with few distractors, performance drops as environments become more cluttered or memory duration increases, providing a clear benchmark for future improvements in robust attention mechanisms.
Conclusion
COG provides a rigorous diagnostic tool for evaluating working memory in AI. While the synthetic nature of the data lacks the visual complexity of real-world video, it successfully isolates the challenges of temporal logic and memory. Future work will involve scaling these architectures to handle more ecologically valid stimuli and exploring even more complex, multi-step algorithmic behaviors.
Summary updated