Abstract
The potential of Vision-Language Models (VLMs) often remains underutilized in handling complex text-based problems, particularly when these problems could benefit from visual representation. Resonating with humans' ability to solve complex text-based problems by (1) creating a visual diagram from the problem and (2) deducing what steps they need to take to solve it, we propose Self-Imagine. We leverage a single Vision-Language Model (VLM) to generate a structured representation of the question using HTML, then render the HTML as an image, and finally use the same VLM to answer the question using both the question and the image. Our approach does not require any additional training data or training. We evaluate our approach on three mathematics tasks and nine general-purpose reasoning tasks using state-of-the-art (LLAVA-1.5 and GEMINI PRO) VLMs. Our approach boosts the performance of LLAVA-1.5 and GEMINI PRO on all math tasks (on average GSM8K: +3.1%; ASDIV: +3.2%; SVAMP: +6.9%) and the majority of the general-purpose reasoning tasks by 3.2% to 6.0% on average.
Introduction
While Vision-Language Models (VLMs) are increasingly powerful, they are often underutilized when handling unimodal (text-only) queries. When a user asks a VLM a math word problem, the model typically defaults to a text-only "Language Model" mode, leaving its sophisticated visual processing capabilities idle.
Humans often solve complex problems by sketching diagrams—a process that helps distill relevant information and map out reasoning steps. The challenge is that most state-of-the-art VLMs (like Gemini Pro or LLaVA) are discriminative or "understanding" models; they aren't natively designed to generate images from scratch like DALL-E. This research introduces "Self-Imagine," a zero-shot technique that forces a VLM to "imagine" a problem visually to enhance its reasoning capabilities.
The Self-Imagine Framework
The researchers propose a three-step pipeline called Self-Imagine:
- HTML Generation: Instead of trying to generate raw pixels, the model is prompted to generate HTML code that represents the structured information in the text query. For example, a math problem about "3 sprints a week" might be turned into a clean HTML table or a series of labeled boxes.
- Rendering: The system uses a standard Python library (
imgkit) to render that HTML into a static image. - Multimodal Reasoning: The original text question and the newly "imagined" image are fed back into the same VLM. The model then performs its final reasoning by looking at both modalities.
This approach is entirely zero-shot/few-shot and requires no fine-tuning. It leverages the model's existing proficiency in code generation as a bridge to visual creation.
Experimental Evaluation
The authors evaluated Self-Imagine across two main categories of tasks:
- Math Word Problems: GSM8K, ASDIV, and SVAMP.
- Symbolic Reasoning (from BIG-Bench Hard): Including tasks like Object Counting, Navigate (spatial directions), Colored Objects, Date Understanding, Penguins in a Table, and Geometric Shapes.
Key Results
The Self-Imagine framework consistently outperformed text-only baselines across various benchmarks.
- Math Performance: On the SVAMP dataset, LLaVA-1.5 saw a +9.3% accuracy boost, while Gemini Pro improved by +4.5%.
- Spatial Reasoning: In the "Navigate" task, LLaVA-1.5 improved by +13.2%, suggesting that even a simple visual layout of directions significantly aids spatial logic.
- General Reasoning: Across the majority of general-purpose reasoning tasks, the approach yielded improvements ranging from 3.2% to 6.0% on average.
Discussion and Insights
- Visual Distillation: The self-generated image acts as a "denoiser." By forcing the model to create a structured HTML representation, it effectively strips away irrelevant "fluff" from word problems, leaving a clean, visual reference point for the final reasoning step.
- The "Code-to-Vision" Bridge: While VLMs struggle to "draw," they are excellent at "coding." Using HTML/SVG as an intermediate step allows the model to use its linguistic/coding strengths to augment its visual weaknesses.
- Failure Modes: If the VLM makes an arithmetic error while writing the HTML, that error becomes "baked into" the image. The model then trusts its own incorrect image over its internal logic, leading to a hallucination. Additionally, HTML is limited for representing temporal changes or complex geometry.
Conclusion
Self-Imagine demonstrates that the visual processing units of VLMs can be effectively activated for text-only tasks through intermediate code generation. By providing a "mind's eye" for these models, we can significantly improve their performance on complex reasoning tasks without the need for additional training data. Future work may explore more sophisticated representational languages like SVG or TikZ to handle more complex geometric and temporal problems.
Summary updated