TL;DR
When faced with text-only math and logic puzzles, Vision-Language Models (VLMs) typically ignore their vision processing pipeline entirely. SELF-IMAGINE fixes this by having a single VLM generate structured HTML code to visually represent a text question, rendering that HTML into an image, and feeding both the image and question back into the model. This zero-shot/few-shot, training-free approach reliably boosts reasoning across math benchmarks and structured symbolic tasks.
What Problem Does This Paper Address?
As general-purpose interfaces transition from pure Large Language Models (LLMs) to multimodal Vision-Language Models (VLMs) like Gemini, these systems increasingly handle unimodal, text-only prompts. When solving complex word problems, humans routinely sketch diagrams to clarify relationships, track variables, and plan steps. Standard VLMs, however, process text queries in purely unimodal text mode—leaving their visual representation engines completely underutilized.
Directly generating images using VLMs is tricky because standard architectures are trained for image comprehension rather than direct image generation, and using an external diffusion model (like DALL-E) adds overhead and disconnects from the reasoning context.
Methods
The authors introduce SELF-IMAGINE, a modular, training-free pipeline executed by a single VLM:
- Structured HTML Generation: Using a 5-shot in-context prompt, the VLM translates a natural language question into structured HTML/CSS code (e.g., tables, variable-value boxes, flowcharts, or SVG shapes).
- Deterministic Rendering: The generated HTML is compiled into a PNG image via the Python library
imgkit. - Multimodal Reasoning: The model receives both the original text prompt and the newly rendered image, allowing its visual encoder and language backbone to co-reason over the grounded problem.
The approach was evaluated on two state-of-the-art VLMs: open-source LLaVA-1.5 (CLIP ViT-L + Vicuna 13B) and proprietary Gemini Pro.
[ Question ] ──(few-shot prompt)──> [ VLM ] ──> [ HTML Code ]
│ (imgkit)
▼
[ Final Answer ] <── [ VLM ] <── [ Question + Rendered Image ]
Datasets
The authors evaluate performance across 12 text-based reasoning benchmarks (no new benchmarks introduced):
- Mathematical Reasoning: GSM8K, ASDIV, and SVAMP.
- Symbolic Reasoning (from BIG-Bench Hard): Object Counting, Navigate, Colored Objects, Date Understanding, Penguins in a Table, Geometric Shapes, and Tracking Shuffled Objects (evaluated across 3, 5, and 7 object variations).
Key Insights and Ideas
- Visuals as De-noising Scratchpads: In math word problems loaded with distractors, visual diagrams act as structural anchors. The HTML generation forces the model to extract core variables into labeled boxes, filtering out extraneous linguistic fluff.
- Repurposing Code Capabilities: Instead of training a visual generative head, the pipeline relies on the model’s pre-trained code generation capabilities to produce programmatic visual layouts (HTML/CSS/SVG).
- Model Specialization in Modality: Different VLMs leverage visual artifacts differently. Gemini Pro (a strong table parser) thrives on tabular representations (Penguins in a Table improved by +6.85%), whereas LLaVA-1.5 benefits on multi-variable tracking (Navigate improved by +13.2%).
- The "Pre-Solving" Pitfall: When generating HTML, the model occasionally attempts mental math prematurely and hardcodes wrong intermediate numbers into the image, biasing its final visual reasoning step toward errors.
Key Results
- Universal Math Improvements: SELF-IMAGINE improved accuracy across all mathematical reasoning tasks for both models:
- SVAMP: +9.30% for LLaVA-1.5 (43.50% $\to$ 52.80%); +4.50% for Gemini Pro (69.50% $\to$ 74.00%).
- GSM8K: +4.62% for LLaVA-1.5 (26.69% $\to$ 31.31%); +1.67% for Gemini Pro (74.37% $\to$ 76.04%).
- ASDIV: +4.49% for LLaVA-1.5 (52.24% $\to$ 56.73%); +2.01% for Gemini Pro (82.01% $\to$ 84.02%).
- Symbolic Reasoning Gains & Losses:
- Strong gains occurred on visualizable tasks like Object Counting (+5.60% LLaVA, +4.40% Gemini), Colored Objects (+5.20% LLaVA, +1.20% Gemini), and Geometric Shapes (+8.70% Gemini).
- However, performance dropped substantially on Tracking Shuffled Objects for Gemini Pro (-13.6% on 3- and 5-object swaps) and Date Understanding for LLaVA-1.5 (-4.80%).
Limitations and Future Work
- Dynamic/Temporal Failure in HTML: Static HTML handles static states, tables, and labeled boxes well, but struggles to represent dynamic operations like sequential state swaps (Tracking Shuffled Objects), insertions, or deletions.
- Context-Length Bottlenecks: For highly complex questions (e.g., question length $>70$ words or long chain-of-thought requirements in GSM8K), HTML generation frequently misses vital constraints, rendering confusing diagrams that harm performance.
- Future Directions: Developing dynamic, step-by-step visual generation mechanisms (such as layered animations or graph-based visual intermediate representations) that do not hallucinate arithmetic steps during diagram creation.
Summary updated