When we ask a modern generative model to draw a sunset, probabilistic pixel prediction works beautifully. The exact placement of a cloud or the precise gradient of orange light does not need to adhere to strict mathematical laws to look correct.
However, this probabilistic approach fails catastrophically when applied to structured visuals—such as scientific charts, geometric diagrams, flowcharts, and user interfaces. If a diffusion model shifts a line in a chart by just a few pixels, or blurs the text in a diagram, the underlying factual data is corrupted. Standard text-to-image models lack spatial precision and logical consistency because they attempt to generate complex topological layouts directly in pixel space.
To solve this, a powerful paradigm has emerged: code-mediated visual synthesis. Instead of translating text descriptions directly into pixels, models generate structured, executable code (such as Python, HTML/CSS, SVG, LaTeX, or Mermaid scripts). This code is then passed to a deterministic rendering engine to produce mathematically precise, perfectly aligned visuals.
[Text Prompt / Data] ──> [LLM Generates Code] ──> [Deterministic Engine] ──> [Precise Visual]
By treating code as an intermediate representation, this approach bridges the gap between linguistic reasoning and exact spatial execution.
Why Code Beats Pixels for Structured Visuals
The core intuition behind this paradigm is that code serves as an unambiguous, symbolic source of truth. When a model writes code, it explicitly defines the logical, mathematical, and spatial relationships of the visual elements before a single pixel is drawn.
1. Eliminating Hallucinations and Spatial Distortion
In direct pixel generation, text rendering is notoriously difficult, and geometric properties like angles or parallel lines are frequently distorted. By generating code instead, these constraints are offloaded to specialized software engines.
For example, Jiaxin Ge demonstrates in AutoPresent: Designing Structured Visuals from Scratch that generating presentation slides via Python code ensures text remains perfectly rendered and layouts remain fully editable, avoiding the text hallucinations common in diffusion models. Similarly, Ishani Mondal et al. show in SciDoc2Diagrammer-MAF: Towards Generation of Scientific Diagrams from Documents guided by Multi-Aspect Feedback Refinement that intermediate code execution handles layout constraints with a level of precision that direct pixel generation cannot match.
2. Mathematical and Coordinate-Based Accuracy
For mathematical and geometric tasks, visual accuracy is non-negotiable. In GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generation, Shihao Cai et al. leverage Wolfram Mathematica code to render geometric diagrams. Because the diagram is generated via exact coordinate-based code, properties like lengths and angles are mathematically precise and perfectly aligned with the problem text.
Generating Flawless Training Data at Scale
Beyond creating visuals for human consumption, the primary engine driving this research is synthetic data generation. To train Vision-Language Models (VLMs) to understand charts or documents, researchers need millions of images paired with accurate textual descriptions or question-answer chains.
Manually annotating these images is expensive and noisy. But if an image is generated from code, the code itself acts as a flawless, auditable ground truth.
┌──> Executable Code ──> Deterministic Renderer ──> Perfect Image
│
[Source Data] ────┼──> Automated Rules ──────────────────────────────> Flawless QA / CoT
Perfect Alignment and Grounding
When an LLM generates a chart using Python plotting code, the exact value of every data point, the color of every bar, and the text of every label are explicitly known.
- Chart Reasoning: Yue Yang et al. developed the CoSyn framework in Scaling Text-Rich Image Understanding via Code-Guided Synthetic Multimodal Data Generation, using text-only LLMs to write Python or HTML code that renders into text-rich images. Because the visual content is bound to the underlying code structure, the system generates high-fidelity training sets completely free of visual hallucinations.
- Chain-of-Thought Synthesis: In Chart-R1: Chain-of-Thought Supervision and Reinforcement for Advanced Chart Reasoner, Lei Chen et al. convert structured tables into plotting code. This allows them to automatically generate multi-step chain-of-thought reasoning paths that are perfectly grounded in the visual elements of the resulting image.
- Web Layouts: Mohammad Amin Bhat et al. introduce a similar pipeline in Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs, pairing language-model-generated HTML/CSS code with automated screenshot tools to guarantee a one-to-one correspondence between structural code and visual layout.
This programmatic generation strategy spans multiple domains, from block diagrams (Shreyanshu Bhushan et al.), to abstract maps and floor plans (Jin, Peng et al.), to mathematical formulas (Libo Qin et al.; Changxin Tian et al.), and document tables (Chaoyi Zhou et al.).
Code-Space Editing and Distillation
The benefits of code-mediated synthesis extend to visual editing and knowledge distillation.
Factually Consistent Visual Editing
Editing a chart or diagram at the pixel level (e.g., changing a value in a bar chart using an image-to-image diffusion model) almost always destroys the semantic integrity of the data. Li Li et al. formalize this challenge in Charts Are Not Images: On the Challenges of Scientific Chart Editing, demonstrating that declarative visualization grammars are required to ensure edits are mathematically precise and auditable.
By shifting the editing task from pixel-space to code-space, models can perform flawless modifications. As Le Zhuo shows in Factuality Matters: When Image Generation and Editing Meet Structured Visuals, synthesizing training data via executable code allows for the creation of precise before-and-after states for editing tasks. Instead of manually altering an image, the model simply manipulates the underlying code parameters, ensuring the modified image remains factually consistent.
[Original Image] ──(Not Pixels!)──> [Modify Code Parameters] ──> [Factually Accurate Edited Image]
Distilling Visual Features
Code can also capture visual-semantic features—like color hierarchies and spatial relationships—more explicitly than raw data tables. In Distill Visual Chart Reasoning Ability from LLMs to MLLMs, Wei He et al. introduce Code-as-Intermediary Translation (CIT). They use Python code as a structured bridge to distill complex visual reasoning from frontier models into smaller, open-source multimodal models, showing that teaching a model to understand the code behind a chart is highly effective for mastering visual logic.
Inference-Time Visual Imagination
While code-mediated synthesis is widely used to generate training data, it can also be leveraged at inference time to help text-only models "think" visually.
In Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination, Syeda Nahida Akter et al. introduce a zero-shot technique where a model solves complex text-only problems by first drawing its own visual aid.
[Complex Text Query] ──> [Model Writes HTML/SVG] ──> [Rendered Image] ──> [Model Reasons Over Image] ──> [Final Answer]
If a text-only query describes a complex spatial arrangement or a mathematical puzzle, the model writes HTML or SVG code to represent the logical components of the query. This code is rendered into an image, and the model then reasons over its own self-generated visual aid to arrive at the correct answer. This technique effectively bridges the gap between a model's linguistic proficiency and its lack of native image generation capabilities, unlocking visual reasoning pathways to solve abstract tasks.
Key Foundational Work
Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination (2024) — contributed 3 concept elements to this direction.
GeoGPT4V: Towards Geometric Multi-modal Large Language Models with Geometric Image Generation (2024) — contributed 2 concept elements to this direction.
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model (2024) — contributed 1 concept elements to this direction.