Abstract
Code has emerged as a precise and executable medium for reasoning and action in the agent era. Yet, progress has largely focused on language-centric tasks, leaving visual-centric coding underexplored. We introduce VCode, a benchmark that reframes multimodal understanding as code generation: given an image, a model must produce Scalable Vector Graphics (SVG) code that preserves symbolic meaning for downstream reasoning. To assess symbolic fidelity, we propose CodeVQA, a novel evaluation protocol where a policy model answers questions over rendered SVGs. We also introduce VCoder, an agentic framework that augments Vision-Language Models (VLMs) through iterative self-revision and external visual tools. VCoder delivers a 12.3-point overall gain over Claude-4-Opus, demonstrating the potential of symbolic visual representations.
Introduction: The Case for Symbolic Visual Coding
While Large Language Models (LLMs) have become expert coders for text-based tasks, their ability to use code for visual reasoning is largely underexplored. Most multimodal models treat images as dense grids of RGB pixels, which are effective for capturing appearance but lack the symbolic abstractions—such as object relationships and precise counts—that facilitate complex reasoning.
We advocate for SVG code as a compact, human-interpretable, and executable visual representation. SVG acts as a "middle ground" or scaffold for reasoning. By requiring a model to write code that describes a scene, we can evaluate exactly what it understands about geometry, layout, and semantics. However, current frontier VLMs reveal a persistent gap between their linguistic coding skills and their visual perception when tasked with generating faithful SVG representations of real-world scenes.
The VCode Benchmark and CodeVQA Evaluation
The VCode benchmark consists of 464 image-question pairs curated from three high-standard benchmarks to cover diverse reasoning domains:
- MM-Vet: General commonsense and everyday semantics.
- MMMU: College-level professional knowledge across various disciplines.
- CV-Bench: Visual-centric perception, specifically 3D depth and spatial relationships.
To evaluate these models, we propose CodeVQA. Instead of using traditional image similarity metrics which may not capture semantic accuracy, CodeVQA requires a model to answer questions about the original image using only the rendered SVG. If the model answers correctly, the generated SVG has successfully preserved the necessary symbolic meaning for reasoning.
The VCoder Framework: Iterative Revision and Visual Tools
To bridge the performance gap in current VLMs, we introduce VCoder, an agentic framework that improves visual coding along two axes:
- Thinking with Revision: An iterative loop where the model renders its generated SVG, compares the render to the original image to identify discrepancies, and refines the code accordingly.
- Acting with Visual Tools: The model is equipped with external "perception toolboxes" (such as object detectors and segmenters). These tools provide structured cues about shapes, text, and spatial locations to guide the code generation process beyond the model's intrinsic capacity.
Experimental Results and Insights
Empirical evaluations show that VCoder achieves an overall score of 54.0, significantly outperforming the base Claude-4-Opus (41.7). Key findings include:
- Tool Impact: Integrating visual tools (location, category, and shape cues) provided a massive 16.6-point improvement on the MM-Vet portion of the benchmark.
- Reasoning vs. Perception: A model's coding quality correlates with its general reasoning ability. However, even models with strong logic often fail at fine-grained visual perception (e.g., distinguishing "near" vs. "far"), making external tools essential.
- Human-VLM Alignment: Human studies indicate that both humans and VLMs perform worse on rendered SVGs than on raw images, but their performance patterns are consistent, suggesting SVG is a valid, albeit lossy, symbolic abstraction.
Conclusion
Despite the gains achieved by VCoder, a significant gap remains between SVG-based reasoning (54.0) and reasoning using raw images (61.7). Models particularly struggle with professional knowledge and complex 3D reasoning. Future work should focus on developing end-to-end vision-language coders trained on larger, more diverse datasets of authored SVG-image pairs to further improve symbolic fidelity.
Summary updated