Assessing GPT4-V on Structured Reasoning Tasks

Mukul Singh, José Cambronero, Sumit Gulwani, Vu Le, Gust VerbruggenarXiv 2023 · OpenReview

Abstract

Multi-modality promises to unlock further uses for large language models. Recently, the state-of-the-art language model GPT-4 was enhanced with vision capabilities. We carry out a prompting evaluation of GPT-4V and five other baselines on structured reasoning tasks, such as mathematical reasoning, visual data analysis, and code generation. We show that visual Chain-of-Thought, an extension of Chain-of-Thought to multi-modal LLMs, yields significant improvements over the vanilla model. We also present a categorized analysis of scenarios where these models perform well and where they struggle, highlighting challenges associated with coherent multimodal reasoning.

Introduction

While GPT-4V represents a massive leap in multimodal AI, it still stumbles on tasks requiring precise "structured reasoning" over visual data. This research introduces visual Chain-of-Thought (v-CoT), a simple yet powerful prompting strategy that forces the model to explicitly describe relevant visual artifacts before reasoning, leading to significant performance gains across math, data analysis, and code generation tasks.

The Perception-Logic Gap

Large Language Models (LLMs) have mastered "thinking step-by-step" for text, but when vision is added to the mix (becoming Large Multimodal Models or LMMs), the reasoning chain often breaks. Even a state-of-the-art model like GPT-4V often fails to connect visual perception with logical deduction. For example, it might see a chart correctly but fail to perform the math, or it might have the right logic but misinterpret a color or a coordinate. This study investigates whether we can bridge this "perception-logic gap" through better prompting rather than expensive fine-tuning.

Methodology: Visual Chain-of-Thought (v-CoT)

The researchers evaluate several prompting strategies, but the centerpiece is their novel visual Chain-of-Thought (v-CoT). Standard multimodal Chain-of-Thought (m-CoT) simply asks the model to "think step-by-step" using the image. In contrast, v-CoT breaks the process into three explicit stages:

  1. Extraction: The model is instructed to first describe only the relevant visual information and artifacts (e.g., "The red line peaks at 50," or "The line passes through point (1,5)").
  2. Reasoning: The model uses those extracted "predicates" to work through the problem logically.
  3. Conclusion: The model states the final answer.
Figure 2: Comparison of standard multimodal Chain-of-Thought (m-CoT) and visual Chain-of-Thought (v-CoT) on a mathematical reasoning task. Red text highlights incorrect reasoning in standard m-CoT, while green text shows correct reasoning in v-CoT.

By separating perception from reasoning, v-CoT prevents the model from "hallucinating" logic based on a vague understanding of the image.

Experimental Setup and Datasets

GPT-4V and several baselines (including Sphinx and InstructBlip) were tested across four diverse domains:

  • MathVista: Mathematical problems requiring visual context (e.g., geometry diagrams).
  • ChartQA: Questions based on data visualizations like bar and line charts.
  • ARC (Abstraction and Reasoning Corpus): A benchmark requiring the model to find patterns in grids of colored blocks.
  • Spider: Traditionally a text-to-SQL dataset, the authors rendered the database tables as images to see if GPT-4V could generate code by "looking" at a table rather than reading its raw text.

Key Insights

  • Perception is the Bottleneck: Many failures were not due to poor logic, but "incorrect grid perception" or "color inferring mistakes." If the model misidentifies a line color in a chart, the most brilliant reasoning in the world will still produce the wrong answer.
  • The "Extraction" Step is a Grounding Mechanism: Forcing the model to list visual facts first ensures the "mental model" the AI uses for reasoning actually matches the pixels in the image.
  • ARC is the "Final Boss": GPT-4V struggled significantly with the ARC (grid) tasks. In some cases, using a different model to describe the grid in text and then letting GPT-4 reason over that text performed better than GPT-4V looking at the image directly, suggesting LMMs still struggle with precise spatial/coordinate-based reasoning.

Results

The v-CoT prompting method consistently outperformed vanilla GPT-4V and other multimodal baselines. As shown in the comparative examples below, v-CoT allows the model to correctly identify data points that zero-shot approaches miss.

Figure 4: Performance comparison between zero-shot GPT-4V and v-CoT on data analysis tasks involving line and bar charts.

Quantitative improvements include:

  • MathVista: v-CoT achieved 49.1% accuracy (vs. 47.5% for vanilla GPT-4V).
  • ChartQA: v-CoT reached 79.2% (vs. 75.6% for vanilla).
  • Spider (Visual SQL): v-CoT hit 86.2% (vs. 84.3% for vanilla).
  • Overall Improvement: v-CoT provided a boost of 1.5 to 9.3 percentage points over standard GPT-4V prompting across the various benchmarks.

Conclusion

While GPT-4V is a powerful tool, its performance on structured reasoning is heavily dependent on how it is prompted to perceive visual data. The success of v-CoT highlights that the current bottleneck in multimodal reasoning is often perception rather than logic. Future work should focus on improving the "spatial awareness" of LMMs and exploring how these models handle multilingual multimodal reasoning.

Summary updated

Abstract

Multi-modality promises to unlock further uses for large language models. Recently, the state-of-the-art language model GPT-4 was enhanced with vision capabilities. We carry out a prompting evaluation of GPT-4V and five other baselines on structured reasoning tasks, such as mathematical reasoning, visual data analysis, and code generation. We show that visual Chain-of-Thought, an extension of Chain-of-Thought to multi-modal LLMs, yields significant improvements over the vanilla model. We also present a categorized analysis of scenarios where these models perform well and where they struggle, highlighting challenges associated with coherent multimodal reasoning.
BibTeX
@misc{singh2023assessing,
  author = {Mukul Singh and José Cambronero and Sumit Gulwani and Vu Le and Gust Verbruggen},
  title = {{Assessing GPT4-V on Structured Reasoning Tasks}},
  year = {2023},
  eprint = {2312.11524},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.11524},
}
Figure