Object Ordering with Bidirectional Matchings for Visual Reasoning

Hao Tan, Mohit Bansal

Researchers from UNC Chapel Hill have introduced a novel approach to visual reasoning, a task that requires an AI to determine the truth value of complex natural language statements describing an image. Their model, BiATT-Pointer, addresses the fundamental mismatch between the sequential nature of language and the unordered "bag of objects" characteristic of visual data.

The Challenge of Visual Reasoning

Visual reasoning extends beyond simple object detection. It requires a model to map linguistic phrases to specific visual entities, understand spatial and numerical relationships, and synthesize information across multiple sub-images. The Cornell Natural Language Visual Reasoning (NLVR) dataset exemplifies these challenges, often requiring a single True/False judgment based on a set of three sub-images.

Examples of visual reasoning tasks from the NLVR dataset, requiring complex spatial and relational understanding.

To verify a statement such as "There is at least one tower with exactly three blocks that has a blue block in the middle," a model must perform several high-level cognitive steps:

  1. Entity Mapping: Identify "tower," "three blocks," and "blue block."
  2. Relational Logic: Interpret "in the middle" and "exactly three."
  3. Sequence Alignment: Reconcile the strictly sequential flow of language with the unordered spatial arrangement of objects in an image.

The BiATT-Pointer Architecture

The authors propose two primary innovations to bridge the vision-language gap: a Reinforcement Learning (RL)-based pointer network for object ordering and a bidirectional attention mechanism.

The BiATT-Pointer architecture, featuring a Pointer Network for object ordering and a Bidirectional Attention (BiATT) module.

1. The Pointer Network (Object Sorter)

Instead of processing visual objects in an arbitrary order, the model employs a Pointer Network trained via RL. This network analyzes both the sentence and the detected objects to determine the optimal sequence in which to "read" the visual elements. By mimicking the way humans scan an image to find specific elements mentioned in a text, the model can leverage powerful sequential processing tools like LSTMs more effectively.

2. Joint Bidirectional Attention (BiATT)

While standard attention mechanisms often flow unidirectionally from language to vision, BiATT establishes a "two-way street." It constructs "object-aware word representations" and "word-aware object representations," ensuring that linguistic terms (e.g., "blue") are conditioned on the specific visual instances present, and vice versa.

3. Multi-View Reasoning

To handle the multi-image nature of the NLVR dataset, the model calculates relevance scores for each sub-image and utilizes a max-pooling layer. This allows the model to identify the most pertinent sub-image for making the final True/False classification.

Results and Performance

The BiATT-Pointer model achieved significant improvements over previous state-of-the-art methods on both structured and raw image versions of the NLVR dataset:

  • Structured Dataset: Accuracy reached 73.9%, a 6.2% absolute improvement over the previous MAXENT baseline.
  • Raw Image Dataset: Accuracy reached 69.7%, outperforming Neural Module Networks by 3.6%.

Ablation studies confirmed the value of the sorting mechanism, showing that the Pointer Network provided a crucial 1.6% performance boost over using bidirectional attention alone.

Discussion and Limitations

Despite its success, the model faces challenges with highly complex spatial reasoning (e.g., "touching any edge") and universal quantification across all sub-images. Because the model relies on max-pooling, it excels at "existence" queries but may struggle with statements requiring a comparison of all three images simultaneously. Future work may explore more flexible pooling strategies to accommodate both existential and universal reasoning.

Summary updated

BibTeX
@inproceedings{tan2018object,
  author = {Hao Tan and Mohit Bansal},
  title = {{Object Ordering with Bidirectional Matchings for Visual Reasoning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=BJ-ZQX-O-S},
}
Figure