When a developer writes HTML for a button or Matplotlib code for a scatter plot, the code is merely a recipe; the visual output is the meal. However, for years, we have evaluated AI models on their ability to memorize the recipe rather than the quality of the resulting dish.
Traditional metrics like BLEU or Edit Distance measure how closely a model's generated code matches a reference string. But as Davit Soselia and colleagues point out, these text-based metrics are "blind" to visual reality. A single missing semicolon might break a script entirely, while a hundred lines of different CSS might produce a visually identical webpage. This phenomenon, often called "representational diversity," means that models are frequently penalized for valid, creative solutions that simply don't match a specific ground-truth string.
To bridge this gap, a new research direction has emerged: visual-in-the-loop learning and evaluation. Instead of comparing text to text, these systems render generated code into an image and use that visual output as the primary source of truth.
The Render and Compare Loop
The core intuition behind this shift is the "round-trip" evaluation. As formalized in the Image2Struct benchmark by Josselin Somerville Roberts, the process follows a simple cycle:
- Input: An image (e.g., a chart, a UI screenshot, or a math formula).
- Generation: The model generates code (e.g., Python, HTML, or LaTeX).
- Execution: The system executes the code to render a new image.
- Comparison: The rendered image is compared to the original input using visual similarity metrics.
This approach allows models to prioritize what matters—the final appearance—over syntactic trivia. For example, in mathematical formula recognition, Fan Wu and others introduced Character Detection Matching (CDM), which renders LaTeX into images to ensure that symbols are spatially and semantically correct, regardless of how the LaTeX code was structured.
Reinforcement Learning with Rendering Rewards
The most powerful application of this idea is in training. Rather than relying solely on Supervised Fine-Tuning (SFT), which often hits a performance plateau, researchers are using Reinforcement Learning (RL) with "rendering rewards."
In the ChartEditor framework, Jianzhe Ma and Yuqi Liu utilize a rendering-based reward to teach models the visual consequences of specific programming instructions. Similarly, Yaozhi Zheng and the ScreenCoder team use pixel-level Mean Squared Error (MSE) between the rendered output and the reference image as a reward signal. This feedback loop forces the model to care about precise alignment and positioning—details that standard code metrics often overlook.
For complex structures like tables, Jun Ling and the Table2LaTeX-RL team demonstrate that this "visual-in-the-loop" strategy allows models to master nested, messy layouts that commercial OCR tools often fail to parse.
Beyond Pixels to Semantic Fidelity
While pixel-level matching is a start, it can be too rigid. A chart with a slightly different font or a slightly lighter shade of blue might be "correct" but have a high MSE. To solve this, researchers are moving toward perceptual and structural rewards.
- Perceptual Similarity: Tom Peterka in ChatVis argues that for scientific visualizations, we must use metrics like CLIP scores or Fréchet Inception Distance (FID) to ensure the meaning of the data is preserved, even if the pixels don't match perfectly.
- Structural Fidelity: In web development, Hongcheng Guo and the IW-Bench team suggest that we should measure "Element Accuracy" and "Layout Accuracy"—checking if the right buttons exist in the right places within the Document Object Model (DOM).
- Aesthetic Rewards: Bang Xiao and the AesCoder-4B team have even begun training models for "code aesthetics," using agentic feedback to ensure generated UIs are not just functional, but visually appealing and free of overlapping text.
Bridging the Perception-Reasoning Gap
Perhaps the most significant insight from this research is that visual-to-code generation acts as a "perceptual check" for Multimodal Large Language Models (MLLMs). Aditya Kanade and the creators of the DYSM benchmark found that models often "reason" correctly about an image while fundamentally misinterpreting its visual elements. By forcing a model to reconstruct an image into a structured format like SVG or TikZ code, we can expose these internal hallucinations.
As Shuhang Chen describes in CogFlow, this process of "knowledge internalization"—where a model must perceive, then internalize the structure, then reason—is essential for preventing "reasoning drift." When a model can successfully render what it sees, it proves it has truly understood the visual world it is trying to describe.