Introduction
While Large Language Models (LLMs) excel at logical and mathematical reasoning through text-based "Chain-of-Thought" (CoT), they frequently struggle with tasks that require spatial or visual "mental imagery." Humans often solve complex spatial problems by sketching them out—think of drawing a map to give directions or doodling a geometric shape to understand a word problem. Whiteboard-of-Thought (WoT) is a simple yet powerful prompting strategy that allows models to write code to draw their intermediate reasoning steps as images, which they then "see" and process to reach a final answer, leading to massive performance gains in spatial and visual tasks.
The Spatial Reasoning Gap
Current Multimodal Large Language Models (MLLMs), despite being trained on billions of images, still struggle with "visual thinking" when the input is purely textual. For example, if you ask a model: "Which lowercase letter is a circle with a vertical line touching it to the right going down?", even top-tier models like GPT-4o often hallucinate and answer "b" instead of "q." This happens because text-based reasoning processes information as a sequence of tokens, which fails to capture the spatial relationships (up, down, left, right) inherent in visual problems. The authors aim to bridge this gap by giving models a metaphorical "whiteboard" to externalize their visual reasoning.
Methodology: The Whiteboard-of-Thought
WoT is a zero-shot prompting method that requires no specialized modules or fine-tuning. The process works in three distinct steps:
- Visualization Code Generation: When given a spatial query, the MLLM is prompted to write Python code (using libraries like
MatplotliborTurtle) that represents the problem visually. - Execution: The code is run in a standard Python interpreter to produce an image.
- Visual Reasoning: The generated image is fed back into the MLLM. The model then uses its native vision capabilities to "look" at its own drawing and provide the final answer.
This approach leverages two existing strengths of modern MLLMs: their high proficiency in writing code and their ability to describe and reason about images.
Evaluation and Key Results
The authors evaluated WoT across several challenging benchmarks, including BIG-Bench ASCII Art (MNIST, Kanji) and Spatial Navigation (grid and non-grid structures). The performance jumps were dramatic, particularly in areas where text-only models previously achieved near-zero accuracy:
- ASCII Kanji Recognition: Jumped from 1.1% (CoT) to 73.8% (WoT).
- ASCII MNIST: Improved from 21.6% to 66.0%.
- Hexagonal Navigation: GPT-4o’s accuracy soared from 8% to 61%.
- 3D ASCII Words: In specific "visual-heavy" fonts, text-only models scored 0%, while WoT achieved 92.1%.
Key Insights and Ideas
- Text is a Spatial Bottleneck: Processing ASCII art as text tokens is like trying to understand a painting by reading a list of hex codes. WoT proves that the "knowledge" of what the shape looks like is often latent in the model, but it needs a visual medium to express and recognize it.
- Code as a Precise Bridge: Unlike text-to-image generative models, which can be imprecise, Python code provides a mathematically precise way for a model to layout spatial coordinates.
- Visual Perception is the New Ceiling: Interestingly, when WoT fails, it’s rarely because the code was wrong. Instead, the model often draws a perfect image but then fails to "see" it correctly—highlighting that MLLM vision modules still have room for improvement.
Limitations and Future Work
The authors identify a few key hurdles, primarily perception errors (responsible for ~75% of failures in the MNIST task) and the reliance on bug-free code. Future work could involve an "agentic" loop where the model looks at its drawing, realizes it made a mistake, and edits the code to fix the visualization before answering.
Summary updated