VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation

Kevin Qinghong Lin, Yuhao Zheng, Hangyu Ran, Dantong Zhu, Dongxing Mao, Linjie Li, Philip Torr, Alex Jinpeng Wang

Abstract

Code has emerged as a precise and executable medium for reasoning and action in the agent era. Yet, progress has largely focused on language-centric tasks, leaving visual-centric coding underexplored. We introduce VCode, a benchmark that reframes multimodal understanding as code generation: given an image, a model must produce Scalable Vector Graphics (SVG) code that preserves symbolic meaning for downstream reasoning. To assess symbolic fidelity, we propose CodeVQA, a novel evaluation protocol where a policy model answers questions over rendered SVGs. We also introduce VCoder, an agentic framework that augments Vision-Language Models (VLMs) through iterative self-revision and external visual tools. VCoder delivers a 12.3-point overall gain over Claude-4-Opus, demonstrating the potential of symbolic visual representations.

Introduction: The Case for Symbolic Visual Coding

While Large Language Models (LLMs) have become expert coders for text-based tasks, their ability to use code for visual reasoning is largely underexplored. Most multimodal models treat images as dense grids of RGB pixels, which are effective for capturing appearance but lack the symbolic abstractions—such as object relationships and precise counts—that facilitate complex reasoning.

We advocate for SVG code as a compact, human-interpretable, and executable visual representation. SVG acts as a "middle ground" or scaffold for reasoning. By requiring a model to write code that describes a scene, we can evaluate exactly what it understands about geometry, layout, and semantics. However, current frontier VLMs reveal a persistent gap between their linguistic coding skills and their visual perception when tasked with generating faithful SVG representations of real-world scenes.

The VCode Benchmark and CodeVQA Evaluation

The VCode benchmark consists of 464 image-question pairs curated from three high-standard benchmarks to cover diverse reasoning domains:

  • MM-Vet: General commonsense and everyday semantics.
  • MMMU: College-level professional knowledge across various disciplines.
  • CV-Bench: Visual-centric perception, specifically 3D depth and spatial relationships.
Distribution of tasks in VCode and an illustration of the CodeVQA evaluation protocol.

To evaluate these models, we propose CodeVQA. Instead of using traditional image similarity metrics which may not capture semantic accuracy, CodeVQA requires a model to answer questions about the original image using only the rendered SVG. If the model answers correctly, the generated SVG has successfully preserved the necessary symbolic meaning for reasoning.

The VCoder Framework: Iterative Revision and Visual Tools

To bridge the performance gap in current VLMs, we introduce VCoder, an agentic framework that improves visual coding along two axes:

  1. Thinking with Revision: An iterative loop where the model renders its generated SVG, compares the render to the original image to identify discrepancies, and refines the code accordingly.
  2. Acting with Visual Tools: The model is equipped with external "perception toolboxes" (such as object detectors and segmenters). These tools provide structured cues about shapes, text, and spatial locations to guide the code generation process beyond the model's intrinsic capacity.
The VCode pipeline: translating pixels to symbolic SVG code and back.

Experimental Results and Insights

Empirical evaluations show that VCoder achieves an overall score of 54.0, significantly outperforming the base Claude-4-Opus (41.7). Key findings include:

  • Tool Impact: Integrating visual tools (location, category, and shape cues) provided a massive 16.6-point improvement on the MM-Vet portion of the benchmark.
  • Reasoning vs. Perception: A model's coding quality correlates with its general reasoning ability. However, even models with strong logic often fail at fine-grained visual perception (e.g., distinguishing "near" vs. "far"), making external tools essential.
  • Human-VLM Alignment: Human studies indicate that both humans and VLMs perform worse on rendered SVGs than on raw images, but their performance patterns are consistent, suggesting SVG is a valid, albeit lossy, symbolic abstraction.

Conclusion

Despite the gains achieved by VCoder, a significant gap remains between SVG-based reasoning (54.0) and reasoning using raw images (61.7). Models particularly struggle with professional knowledge and complex 3D reasoning. Future work should focus on developing end-to-end vision-language coders trained on larger, more diverse datasets of authored SVG-image pairs to further improve symbolic fidelity.

Summary updated

Abstract

Code has emerged as a precise and executable medium for reasoning and action in the agent era. Yet, progress has largely focused on language-centric tasks such as program synthesis and debugging, leaving visual-centric coding underexplored. Inspired by how humans reason over sketches, we advocate SVG code as a compact, interpretable, and executable visual representation. We introduce VCode, a benchmark that reframes multimodal understanding as code generation: given an image, a model must produce SVG that preserves symbolic meaning for downstream reasoning. VCode covers three domains - general commonsense (MM-Vet), professional disciplines (MMMU), and visual-centric perception (CV-Bench). To assess symbolic fidelity, we propose CodeVQA, a novel evaluation protocol in which a policy model answers questions over rendered SVGs; correct answers indicate faithful symbolic preservation. Empirically, frontier VLMs struggle to generate faithful SVGs, revealing a persistent gap between language-centric and visual-centric coding. To close this gap, we introduce VCoder, an agentic framework that augments VLMs along two axes: (i) Thinking with Revision, which iteratively analyzes discrepancies and refines SVG code; and (ii) Acting with Visual Tools, where detectors and parsers supply structured cues such as objects, shapes, and text beyond the model's intrinsic capacity. Across benchmarks, frontier VLMs with strong reasoning capabilities score well overall yet remain limited in professional knowledge and 3D reasoning. VCoder delivers a 12.3-point overall gain over the top-performing Claude-4-Opus. Human studies show that both humans and VLMs perform worse on rendered SVGs, their consistency reveals the promise of symbolic visual representation. The benchmark and code are available at https://github.com/CSU-JPG/VCode.
BibTeX
@misc{lin2025vcode,
  author = {Kevin Qinghong Lin and Yuhao Zheng and Hangyu Ran and Dantong Zhu and Dongxing Mao and Linjie Li and Philip Torr and Alex Jinpeng Wang},
  title = {{VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation}},
  year = {2025},
  eprint = {2511.02778},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.02778},
}
Figure