Abstract
Visual reasoning with compositional natural language instructions, e.g., based on the newly-released Cornell Natural Language Visual Reasoning (NLVR) dataset, is a challenging task, where the model needs to have the ability to create an accurate mapping between the diverse phrases and the several objects placed in complex arrangements in the image. Further, this mapping needs to be processed to answer the question in the statement given the ordering and relationship of the objects across three similar images. In this paper, we propose a novel end-to-end neural model for the NLVR task, where we first use joint bidirectional attention to build a two-way conditioning between the visual information and the language phrases. Next, we use an RL-based pointer network to sort and process the varying number of unordered objects (so as to match the order of the statement phrases) in each of the three images and then pool over the three decisions. Our model achieves strong improvements (of 4-6% absolute) over the state-of-the-art on both the structured representation and raw image versions of the dataset.
Introduction: The Challenges of Visual Reasoning
Visual reasoning requires a model to interpret complex natural language statements alongside corresponding visual data to determine the statement's truth value. The Cornell Natural Language Visual Reasoning (NLVR) dataset highlights three primary hurdles in this domain:
- Compositional Language: Statements often involve intricate spatial relationships, counts, and specific attributes (e.g., "There is a black block attached to a yellow block that is attached to a blue block").
- Unordered Object Sets: In structured data versions, objects are provided as an arbitrary list. Standard sequence models like LSTMs struggle when the input order lacks inherent meaning.
- Multi-Image Reasoning: Each task involves three sub-images. The model must determine if a statement applies to the set collectively, often requiring existential logic (e.g., "At least one tower...").
The BiATT-Pointer Model
To address these challenges, we propose the BiATT-Pointer model, which integrates three core components:
1. The Pointer Network (Object Ordering)
To solve the "unordered objects" problem, the model employs a pointer network to learn an optimal permutation of objects. Since no ground-truth order exists, the network is trained via Reinforcement Learning (Policy Gradient). It receives a reward based on classification accuracy, effectively learning to "scan" objects in an order that aligns with the linguistic structure of the query.
2. Joint Bidirectional Attention (BiATT)
Once objects are ordered, the model establishes a two-way bridge between text and visual features. Unlike unidirectional models, BiATT computes both word-to-object and object-to-word attention. This creates joint representations where every word is contextualized by relevant objects and vice versa, facilitating a deeper understanding of relationships like "touching" or "above."
3. Max-Pooling over Sub-Images
To handle the multi-image structure, the model calculates a truth score for each sub-image individually and applies a max-pooling layer. This is particularly effective for NLVR tasks where statements often concern the existence of a pattern in at least one sub-image.
Experimental Results
The BiATT-Pointer model established a new state-of-the-art on both versions of the NLVR dataset:
- Structured Dataset: Achieved 73.9% accuracy, representing a 6.2% absolute improvement over the previous best end-to-end model (MAXENT).
- Raw Image Dataset: Achieved 69.7% accuracy, outperforming Neural Module Networks (NMN) by 3.6%.
- Ablation Studies: Removing either the pointer network or the bidirectional attention mechanism resulted in significant performance drops, confirming that both components are essential for effective visual reasoning.
Limitations and Future Work
While the model shows significant gains, it still faces difficulties with rare high-level phrases (e.g., "touching the base") and complex multi-image logic that requires coordination across sub-images (e.g., "Exactly two sub-images contain..."). Future research may explore more sophisticated pooling methods, such as LSTM-fusion, to handle complex logical quantifiers across image sets.
Summary updated