Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities

Sachit Menon, Richard S. Zemel, Carl VondrickEMNLP 2024 · DOI · OpenReview

Introduction

While Large Language Models (LLMs) excel at logical and mathematical reasoning through text-based "Chain-of-Thought" (CoT), they frequently struggle with tasks that require spatial or visual "mental imagery." Humans often solve complex spatial problems by sketching them out—think of drawing a map to give directions or doodling a geometric shape to understand a word problem. Whiteboard-of-Thought (WoT) is a simple yet powerful prompting strategy that allows models to write code to draw their intermediate reasoning steps as images, which they then "see" and process to reach a final answer, leading to massive performance gains in spatial and visual tasks.

The Spatial Reasoning Gap

Current Multimodal Large Language Models (MLLMs), despite being trained on billions of images, still struggle with "visual thinking" when the input is purely textual. For example, if you ask a model: "Which lowercase letter is a circle with a vertical line touching it to the right going down?", even top-tier models like GPT-4o often hallucinate and answer "b" instead of "q." This happens because text-based reasoning processes information as a sequence of tokens, which fails to capture the spatial relationships (up, down, left, right) inherent in visual problems. The authors aim to bridge this gap by giving models a metaphorical "whiteboard" to externalize their visual reasoning.

Methodology: The Whiteboard-of-Thought

WoT is a zero-shot prompting method that requires no specialized modules or fine-tuning. The process works in three distinct steps:

  1. Visualization Code Generation: When given a spatial query, the MLLM is prompted to write Python code (using libraries like Matplotlib or Turtle) that represents the problem visually.
  2. Execution: The code is run in a standard Python interpreter to produce an image.
  3. Visual Reasoning: The generated image is fed back into the MLLM. The model then uses its native vision capabilities to "look" at its own drawing and provide the final answer.
Comparison between standard Chain-of-Thought (text-only) and Whiteboard-of-Thought (code-to-image) for a spatial letter identification task.

This approach leverages two existing strengths of modern MLLMs: their high proficiency in writing code and their ability to describe and reason about images.

Evaluation and Key Results

The authors evaluated WoT across several challenging benchmarks, including BIG-Bench ASCII Art (MNIST, Kanji) and Spatial Navigation (grid and non-grid structures). The performance jumps were dramatic, particularly in areas where text-only models previously achieved near-zero accuracy:

  • ASCII Kanji Recognition: Jumped from 1.1% (CoT) to 73.8% (WoT).
  • ASCII MNIST: Improved from 21.6% to 66.0%.
  • Hexagonal Navigation: GPT-4o’s accuracy soared from 8% to 61%.
  • 3D ASCII Words: In specific "visual-heavy" fonts, text-only models scored 0%, while WoT achieved 92.1%.

Key Insights and Ideas

  • Text is a Spatial Bottleneck: Processing ASCII art as text tokens is like trying to understand a painting by reading a list of hex codes. WoT proves that the "knowledge" of what the shape looks like is often latent in the model, but it needs a visual medium to express and recognize it.
  • Code as a Precise Bridge: Unlike text-to-image generative models, which can be imprecise, Python code provides a mathematically precise way for a model to layout spatial coordinates.
  • Visual Perception is the New Ceiling: Interestingly, when WoT fails, it’s rarely because the code was wrong. Instead, the model often draws a perfect image but then fails to "see" it correctly—highlighting that MLLM vision modules still have room for improvement.

Limitations and Future Work

The authors identify a few key hurdles, primarily perception errors (responsible for ~75% of failures in the MNIST task) and the reliance on bug-free code. Future work could involve an "agentic" loop where the model looks at its drawing, realizes it made a mistake, and edits the code to fix the visualization before answering.

Summary updated

Abstract

When presented with questions involving visual thinking, humans naturally switch reasoning modalities, often forming mental images or drawing visual aids. Large language models have shown promising results in arithmetic and symbolic reasoning by expressing intermediate reasoning in text as a chain of thought, yet struggle to extend this capability to answer text queries that are easily solved by visual reasoning, even with extensive multimodal pretraining. We introduce a simple method, whiteboard-of-thought prompting, to unlock the visual reasoning capabilities of multimodal large language models across modalities. Whiteboard-of-thought prompting provides multimodal large language models with a metaphorical ‘whiteboard’ to draw out reasoning steps as images, then returns these images back to the model for further processing. We find this can be accomplished with no demonstrations or specialized modules, instead leveraging models’ existing ability to write code with libraries such as Matplotlib and Turtle. This simple approach shows state-of-the-art results on four difficult natural language tasks that involve visual and spatial reasoning. We identify multiple settings where GPT-4o using chain-of-thought fails dramatically, including more than one where it achieves 0% accuracy, while whiteboard-of-thought enables up to 92% accuracy in these same settings. We present a detailed exploration of where the technique succeeds as well as its sources of error.
BibTeX
@inproceedings{menon2024whiteboard,
  author = {Sachit Menon and Richard S. Zemel and Carl Vondrick},
  title = {{Whiteboard-of-Thought: Thinking Step-by-Step Across Modalities}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1117},
}
Figure