TL;DR
VCode reframes multimodal understanding as visual code generation: instead of treating images as dense pixel grids, vision-language models (VLMs) must convert natural images into Scalable Vector Graphics (SVG) code that preserves essential symbolic structure. To benchmark fidelity, the authors evaluate whether downstream visual question answering (VQA) succeeds on the rendered SVGs. They introduce VCoder, an agentic framework combining tool use (detection, segmentation, OCR) with iterative visual revision, yielding an +8.7-point overall improvement over Claude-4-Opus.
What Problem Does This Paper Address?
Current multimodal models typically process natural images as dense RGB pixel grids or patches. While rich in visual texture, raw pixels lack compact symbolic abstraction. In contrast, humans naturally conceptualize visual scenes as abstract, structured sketches highlighting spatial relationships and boundaries.
Although multimodal coding benchmarks exist (e.g., generating HTML/CSS from UI mockups or Matplotlib code from charts), they focus almost exclusively on synthetic assets. Whether VLMs can translate complex, real-world natural scenes into compact, executable code (specifically SVG) while retaining semantic meaning has remained unexplored.
Methods
The paper introduces an evaluation paradigm and an agentic model architecture:
- VCode & CodeVQA Benchmark Protocol: Given an input image $V$, a coder model generates SVG code $C$. Rendering $C$ produces $\tilde{V}$. Instead of relying solely on pixel/embedding similarity (e.g., SigLIP-2 cosine distance), the primary metric is CodeVQA: a policy model (such as GPT-4o-mini) answers questions about the original scene using only the rendered SVG image $\tilde{V}$. A correct answer confirms that essential semantic information was faithfully preserved in code.
- VCoder Framework: Directly generating SVG from natural images struggles with long context windows, visual blindness during text output, and fine-grained localization. VCoder augments VLMs along two axes:
- Thinking with Revision: An iterative test-time loop where the model inspects intermediate renderings $\tilde{V}^{(t)}$, identifies discrepancies against the target image $V$, and rewrites the SVG code conditioned on the visual feedback.
- Acting with Visual Tools: Injects structured visual metadata directly into the coding prompt using external models:
- Localization & Category: Florence-2 provides bounding boxes and semantic labels.
- Shape: SAM-2 extracts irregular segmentation contours, simplified into compact polygon paths.
- Text: OpenOCR detects and transcribes scene text directly into native SVG
<text>elements.
Input Image ──► Tool Perception (Detection, SAM-2, OCR)
│
▼
Initial SVG Generation
│
▼
Render & Compare (Thinking) ◄──┐
│ │ (Iterative Revision)
▼ │
Refined SVG Output ────────────┘
Datasets
The authors curated VCode, a dataset containing 464 single-image VQA instances sampled across three distinct evaluation domains:
- General Commonsense: The entire MM-Vet benchmark (218 pairs) covering spatial awareness, math, and general reasoning.
- Professional Knowledge: A filtered single-image subset of the MMMU dev set (146 pairs) spanning diploma-level disciplines.
- Visual-Centric Perception: A stratified sample of CV-Bench (100 pairs) balancing spatial relations (20), object counting (20), depth ordering (30), and relative distance (30).
Key Insights and Ideas
- Human vs. VLM Abstraction Divergence: On original raw images, state-of-the-art VLMs significantly outscored humans (75.5% for GLM-4.5V vs. 51.6% for humans). However, when answering questions over rendered SVGs, model accuracy dropped sharply while human accuracy rose to 56.9%—beating GPT-4o-mini and Claude-4-Opus. This suggests current VLMs over-rely on low-level appearance cues, whereas humans excel at reasoning over symbolic visual abstractions.
- Language as an Intermediate Scaffold: Generating SVG directly from images (
Img2SVG, 42.5% overall) performs worse than generating an intermediate text description first (Img2Text2Svg, 46.4%), proving that current coders still heavily rely on linguistic representations to bridge visual concepts to code. - SVG Specialists Fail on Natural Scenes: Models specialized in vector graphics (such as StarVector-8B, which scores only 8.9% overall) struggle on natural images, underscoring the gap between clean icon/logo datasets and the messiness of real-world scenes.
Key Results
- Baseline Coders: Among out-of-the-box models, closed-source models with strong reasoning outperformed open-source models. GPT-5 established the strongest native baseline with a 72.1 SigLIP score and 45.7% overall CodeVQA (trailing the raw image upper bound of 55.8%).
- VCoder Gains: VCoder (built on Claude-4-Opus) improved overall CodeVQA accuracy from 41.8% to 50.5% (+8.7 points), significantly narrowing the gap to raw-image performance.
- Domain Breakdown for VCoder:
- MM-Vet: Improved from 37.5% to 54.1% (+16.6 points).
- MMMU: Improved from 43.2% to 50.0% (+6.8 points).
- CV-Bench: Modest adjustment on 3D perception, settling at 43.6%.
- Ablation Takeaways: Adding bounding boxes, shapes, and OCR incrementally improved performance from 37.5% to 52.4% on MM-Vet, with test-time revision providing an additional boost to 54.1%.
Limitations and Future Work
- Perception Bottlenecks in 3D: Performance on CV-Bench (relative distance and depth ordering) remains low for all models, showing that standard 2D SVG representations struggle to capture 3D spatial geometry.
- Knowledge-Intensive Gaps: The "Knowledge" subcategory remains the lowest-scoring area across models, as reconstructing historical artifacts or scientific diagrams requires deep factual retrieval combined with visual synthesis.
- Pipeline Modularity: VCoder currently relies on a multi-stage agent pipeline with external perception tools. The authors highlight the need for end-to-end vision-language models natively trained on large-scale natural image-to-SVG pairs.
Summary updated