VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation

Kevin Qinghong Lin, Yuhao Zheng, Hangyu Ran, Dongxing Mao, Dantong Zhu, Linjie Li, Philip Torr, Alex Jinpeng WangSubmission to ICLR 2026 · Rejected

TL;DR

VCode reframes multimodal understanding as visual code generation: instead of treating images as dense pixel grids, vision-language models (VLMs) must convert natural images into Scalable Vector Graphics (SVG) code that preserves essential symbolic structure. To benchmark fidelity, the authors evaluate whether downstream visual question answering (VQA) succeeds on the rendered SVGs. They introduce VCoder, an agentic framework combining tool use (detection, segmentation, OCR) with iterative visual revision, yielding an +8.7-point overall improvement over Claude-4-Opus.

What Problem Does This Paper Address?

Current multimodal models typically process natural images as dense RGB pixel grids or patches. While rich in visual texture, raw pixels lack compact symbolic abstraction. In contrast, humans naturally conceptualize visual scenes as abstract, structured sketches highlighting spatial relationships and boundaries.

Although multimodal coding benchmarks exist (e.g., generating HTML/CSS from UI mockups or Matplotlib code from charts), they focus almost exclusively on synthetic assets. Whether VLMs can translate complex, real-world natural scenes into compact, executable code (specifically SVG) while retaining semantic meaning has remained unexplored.

Methods

The paper introduces an evaluation paradigm and an agentic model architecture:

  • VCode & CodeVQA Benchmark Protocol: Given an input image $V$, a coder model generates SVG code $C$. Rendering $C$ produces $\tilde{V}$. Instead of relying solely on pixel/embedding similarity (e.g., SigLIP-2 cosine distance), the primary metric is CodeVQA: a policy model (such as GPT-4o-mini) answers questions about the original scene using only the rendered SVG image $\tilde{V}$. A correct answer confirms that essential semantic information was faithfully preserved in code.
  • VCoder Framework: Directly generating SVG from natural images struggles with long context windows, visual blindness during text output, and fine-grained localization. VCoder augments VLMs along two axes:
  • Thinking with Revision: An iterative test-time loop where the model inspects intermediate renderings $\tilde{V}^{(t)}$, identifies discrepancies against the target image $V$, and rewrites the SVG code conditioned on the visual feedback.
  • Acting with Visual Tools: Injects structured visual metadata directly into the coding prompt using external models:
  • Localization & Category: Florence-2 provides bounding boxes and semantic labels.
  • Shape: SAM-2 extracts irregular segmentation contours, simplified into compact polygon paths.
  • Text: OpenOCR detects and transcribes scene text directly into native SVG <text> elements.
Input Image ──► Tool Perception (Detection, SAM-2, OCR)
                      │
                      ▼
               Initial SVG Generation
                      │
                      ▼
            Render & Compare (Thinking) ◄──┐
                      │                    │ (Iterative Revision)
                      ▼                    │
            Refined SVG Output ────────────┘

Datasets

The authors curated VCode, a dataset containing 464 single-image VQA instances sampled across three distinct evaluation domains:

  • General Commonsense: The entire MM-Vet benchmark (218 pairs) covering spatial awareness, math, and general reasoning.
  • Professional Knowledge: A filtered single-image subset of the MMMU dev set (146 pairs) spanning diploma-level disciplines.
  • Visual-Centric Perception: A stratified sample of CV-Bench (100 pairs) balancing spatial relations (20), object counting (20), depth ordering (30), and relative distance (30).

Key Insights and Ideas

  • Human vs. VLM Abstraction Divergence: On original raw images, state-of-the-art VLMs significantly outscored humans (75.5% for GLM-4.5V vs. 51.6% for humans). However, when answering questions over rendered SVGs, model accuracy dropped sharply while human accuracy rose to 56.9%—beating GPT-4o-mini and Claude-4-Opus. This suggests current VLMs over-rely on low-level appearance cues, whereas humans excel at reasoning over symbolic visual abstractions.
  • Language as an Intermediate Scaffold: Generating SVG directly from images (Img2SVG, 42.5% overall) performs worse than generating an intermediate text description first (Img2Text2Svg, 46.4%), proving that current coders still heavily rely on linguistic representations to bridge visual concepts to code.
  • SVG Specialists Fail on Natural Scenes: Models specialized in vector graphics (such as StarVector-8B, which scores only 8.9% overall) struggle on natural images, underscoring the gap between clean icon/logo datasets and the messiness of real-world scenes.

Key Results

  • Baseline Coders: Among out-of-the-box models, closed-source models with strong reasoning outperformed open-source models. GPT-5 established the strongest native baseline with a 72.1 SigLIP score and 45.7% overall CodeVQA (trailing the raw image upper bound of 55.8%).
  • VCoder Gains: VCoder (built on Claude-4-Opus) improved overall CodeVQA accuracy from 41.8% to 50.5% (+8.7 points), significantly narrowing the gap to raw-image performance.
  • Domain Breakdown for VCoder:
    • MM-Vet: Improved from 37.5% to 54.1% (+16.6 points).
  • MMMU: Improved from 43.2% to 50.0% (+6.8 points).
  • CV-Bench: Modest adjustment on 3D perception, settling at 43.6%.
  • Ablation Takeaways: Adding bounding boxes, shapes, and OCR incrementally improved performance from 37.5% to 52.4% on MM-Vet, with test-time revision providing an additional boost to 54.1%.

Limitations and Future Work

  • Perception Bottlenecks in 3D: Performance on CV-Bench (relative distance and depth ordering) remains low for all models, showing that standard 2D SVG representations struggle to capture 3D spatial geometry.
  • Knowledge-Intensive Gaps: The "Knowledge" subcategory remains the lowest-scoring area across models, as reconstructing historical artifacts or scientific diagrams requires deep factual retrieval combined with visual synthesis.
  • Pipeline Modularity: VCoder currently relies on a multi-stage agent pipeline with external perception tools. The authors highlight the need for end-to-end vision-language models natively trained on large-scale natural image-to-SVG pairs.

Summary updated

Abstract

Code has emerged as a precise, executable medium for reasoning and action in the agent era. Yet progress has largely focused on linguistic-centric tasks, such as program synthesis and debugging, leaving visual-centric coding underexplored. Conventional image representations rely on dense RGB pixels that capture appearance but provide limited symbolic abstraction. Inspired by how humans reason over sketches, we advocate SVG code as a compact, interpretable, and executable visual representation. We introduce VCode, a benchmark that reframes multimodal understanding as code generation: given an image, a model must produce SVG that preserves symbolic meaning for downstream reasoning. VCode covers three challenging domains—general commonsense (MM-Vet), professional disciplines (MMMU), and visual-centric perception (CV-Bench). To assess symbolic fidelity, we propose CodeVQA, a novel evaluation protocol in which a policy model answers questions over rendered SVG; correct answers indicate faithful symbolic preservation. Empirically, frontier VLMs struggle to generate faithful SVGs, revealing a persistent gap between language-centric and visual-centric coding. To close this gap, we introduce VCoder, an agentic framework that augments VLMs along two axes: (i) Thinking with Revision, which iteratively analyzes discrepancies and refines SVG code; and (ii) Acting with Visual Tools, where detectors and parsers supply structured cues (objects, shapes, text) beyond intrinsic model capacity. Across benchmarks, frontier VLMs with strong reasoning score well overall yet remain limited on professional knowledge and 3D reasoning; VCoder delivers a +8.7 point overall gain over the top-performing Claude-4-Opus. Human studies further show that although VLMs score higher on raw images, humans are more robust on rendered SVGs—underscoring symbolic visual coding as a promising paradigm for human-like multimodal intelligence.
BibTeX
@misc{lin2025vcode,
  author = {Kevin Qinghong Lin and Yuhao Zheng and Hangyu Ran and Dongxing Mao and Dantong Zhu and Linjie Li and Philip Torr and Alex Jinpeng Wang},
  title = {{VCode: a Multimodal Coding Benchmark with SVG as Symbolic Visual Representation}},
  year = {2025},
  eprint = {2511.02778},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.02778},
}
Figure