Abstract
Multimodal Large Language Models (MLLMs) often struggle with visual reasoning due to the high density of redundant information in raw images. We present VAT (Thinking with Visual Abstract), a prompting paradigm that supplements original images with simplified visual abstracts, such as sketches or edge maps. VAT filters out visual noise, allowing models to focus on structural essentials. Our results show that VAT outperforms standard Chain-of-Thought (CoT) methods in both accuracy and efficiency across multiple benchmarks, including a new "Odd-One-Out" task.
The Challenge of Visual Redundancy
While images are rich in information, they are often too rich for efficient machine reasoning. Raw pixels contain immense amounts of redundant data—textures, lighting, and background clutter—that can distract a model from core tasks like counting objects or understanding spatial relationships.
Existing solutions generally rely on Verbal Chain-of-Thought (CoT), which is often verbose and expensive, or Visual Tools that use external agents to segment images, adding significant computational overhead. VAT seeks to replicate human cognitive strategies by mentally simplifying complex scenes into core concepts before reasoning.
Methodology: Thinking with Visual Abstracts
The VAT framework involves a two-step process:
- Visual Abstraction: Input images are transformed into "abstracts" using techniques ranging from simple code-based methods (Canny edge detection, Binarization) to deep-learning-based styles (PhotoSketch, OpenSketch).
- Dual-Image Prompting: The MLLM is prompted with both the original raw image and the visual abstract. The abstract serves as a "perceptual guide," helping the model focus on contours and structural features without requiring lengthy intermediate text.
Key Insights and Results
The study evaluates VAT across a wide array of benchmarks, including BLINK, CoSpace, and a new Odd-One-Out benchmark where models must identify a single different object in a grid.
- Performance Gains: VAT achieved an average gain of +2.21% over the GPT-5 baseline, significantly outperforming the +0.96% gain seen with standard CoT. The benefits were most pronounced in larger models; GPT-4o saw a 15.33% improvement over its CoT baseline in certain tasks.
- Efficiency: Because VAT doesn't require the model to output hundreds of intermediate "thought" tokens, it is much faster than CoT or tool-based agents. On GPT-4o, VAT required only 0.28x the runtime of tool-using methods and consumed 90% fewer tokens.
- Style Matters: Not all abstracts are equal. OpenSketch, which retains geometric depth and salient edges, generally performed best. Interestingly, combining multiple abstract styles often decreased performance by re-introducing redundancy.
Conclusion
VAT demonstrates that "less is more" in visual reasoning. By providing a denoised structural guide, MLLMs can achieve higher accuracy with lower computational costs. While the abstraction process can sometimes strip away fine-grained textures needed for specific tasks, the overall paradigm shift from verbal to visual abstract thinking represents a significant step forward in multimodal model efficiency.
Summary updated