Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding

Haolin Yang, Wenshan Wu, Jin Jiang, Yu Cheng, Yan Xia, Ting Song, Yangyu Huang, Zhiliang Peng, Wenhui Wang, Hangbo Bao, Yuzhe Zhang, Wen-Da Wei, Tengchao Lv, Li Dong, Lei Cui, Furu WeiarXiv 2026 · arXiv

TL;DR

The Vision-as-a-Tool (VaaT) paradigm shifts visual reasoning from an "intrinsic" capability of a model to an external, callable utility. By using the VHarness framework—a set of five atomic visual actions—text-only agents can now outperform multimodal ones on complex coding tasks, while multimodal agents can break through their own perceptual bottlenecks to achieve unprecedented accuracy in generating HTML, SVG, and 3D models.

What Problem Does This Paper Address?

Most current multimodal agents operate on a "Thinking-with-Images" paradigm. In this setup, the agent’s reasoning and perception are tightly coupled; the model "sees" with its own internal weights. This creates two major hurdles:

  1. Perceptual Bottlenecks: An agent’s performance is strictly capped by its native vision. If the model can't perceive a fine detail or a complex geometric relationship, it cannot code for it.
  2. Exclusion of Text-Only Models: Powerful text-only LLMs are effectively "blind" and cannot participate in agentic coding tasks that require visual feedback, such as webpage reconstruction or 3D modeling.

The authors argue that perception should be decoupled from reasoning, much like how a software engineer uses external debuggers or shell utilities rather than relying solely on memory.

Methods

The authors introduce VHarness, a framework that reframes a general-purpose Multimodal Large Language Model (MLLM)—in this study, Gemini-3.8-Flash—as a suite of external visual utilities. This is instantiated via the Model Context Protocol (MCP), allowing any agent to invoke five "atomic actions":

  • transcribe: Generates a high-level structural skeleton or layout of a target image.
  • grep: Locates specific visual elements and returns their normalized bounding boxes.
  • inspect: Provides fine-grained details about a specific region (e.g., typography, exact colors, or geometric properties).
  • align: A visual assertion tool that checks if a rendered artifact matches a specific text description.
  • diff: Compares the target image with the agent's current rendered draft, localizing discrepancies to guide the next round of edits.
Figure 1. Overview of VHarness, an instantiation of the Vision-as-a-Tool (VaaT) paradigm. Given a target visual input across Image-to-HTML, Image-to-SVG, and Image-to-3D tasks, a multimodal or text-only coding agent iteratively refines rendered Working Artifacts toward the final Code Artifacts. Rather than following a fixed workflow, the coding agent chooses among intrinsic vision when available (dashed blue), code writing and editing (green), and external visual utilities (purple) on demand, as shown in the Interleaved Trajectory.

Rather than a fixed pipeline, the agent autonomously decides which tool to call during an Interleaved Trajectory, iteratively refining its code until the visual output matches the target.

Datasets

The researchers evaluated VHarness on three demanding "vision-to-code" tasks, using 100 test cases for each:

  • Image-to-HTML: Curated complex webpage designs, including editorial magazines and promotional posters with irregular text wrapping.
  • Image-to-SVG: Graphic designs requiring the synthesis of Bézier paths, gradients, and multi-layered vector primitives.
  • Image-to-3D: Randomly sampled cases from 3DCodeBench, where agents must generate Blender Python scripts to create 3D meshes from 2D views.

The authors noted that existing benchmarks like Design2Code were too simple for modern MLLMs, necessitating these more "realistic and challenging" curated sets.

Key Insights and Ideas

  • Decoupling is Power: The most striking insight is that reasoning and perception do not need to reside in the same model. By offloading "vision" to a specialized tool, the "brain" (the coding agent) can focus entirely on logic and syntax.
  • The "Blind" Can Lead: A text-only agent (GLM-5.2) equipped with VHarness was able to compete with and often beat native multimodal models like GPT-5.5. This suggests that high-quality tool-use can compensate for a lack of native senses.
  • Iterative Refinement vs. One-Shot: VHarness enables a "closed-loop" system. The diff tool, in particular, acts as a visual debugger, allowing the agent to see exactly where its code failed to reproduce the target.
  • Structured Feedback Matters: Simply giving an agent "freestyle" access to an MLLM is less effective than providing the structured, atomic actions of VHarness. The constraints of the tools (like returning JSON-formatted bounding boxes) help the agent process visual information more reliably.

Key Results

The quantitative gains were consistent across all models and tasks. Using VHarness improved every metric, including 2D perceptual fidelity (SigLIP2, DINOv2) and 3D geometric accuracy (Uni3D, Chamfer distance).

  • Multimodal Boost: GPT-5.5 with VHarness outperformed the stronger GPT-5.6-sol baseline in Image-to-3D tasks.
  • Text-Only Breakthrough: The text-only GLM-5.2 + VHarness outperformed the multimodal Claude Opus 4.8 across all three tasks.
  • Qualitative Superiority: As seen in Figure 2, VHarness allowed agents to capture complex details that baselines missed, such as organic branching in 3D trees and curved text wrapping in HTML.
Qualitative comparisons showing VHarness's superior ability to capture fine details and complex geometry.

Limitations and Future Work

While highly effective, the authors acknowledge several limitations:

  • Static Targets: The current study focuses on static images and 3D objects. It does not yet address dynamic, multi-state interactions (like clicking a button in a web app) or temporal 3D scenes (video).
  • Inference Cost: Calling an external MLLM multiple times per task increases token consumption and latency, though the authors argue the quality gains justify the cost.
  • Tool Evolution: Future work could expand the atomic actions to include more specialized domain knowledge, such as accessibility checking or performance profiling for the generated code.

Summary updated

Abstract

Multimodal agentic coding requires robust visual reasoning to analyze observations and evaluate generated artifacts. Current frameworks largely rely on a Thinking-with-Images paradigm, where agents depend on their intrinsic vision to execute visually grounded tasks. However, this architectural coupling introduces a fundamental bottleneck. Agent performance remains strictly bounded by internal perceptual limitations. Furthermore, text-only agents lack native perception. Without specialized tools, they cannot perform visual reasoning. To overcome these constraints, we introduce the Vision-as-a-Tool (VaaT) paradigm. Prior visual tool-use merely manipulates images for an agent's intrinsic vision to read. In contrast, VaaT reframes general-purpose multimodal large language models (MLLMs) as callable external utilities. As a result, text-only agents can completely offload visual reasoning during agentic execution. Meanwhile, vision-capable agents can dynamically augment their native perception on demand. We instantiate this paradigm in VHarness, a unified framework driven by five atomic actions. We comprehensively evaluate VHarness across Image-to-HTML, Image-to-SVG, and Image-to-3D generation tasks. Experimental results demonstrate that VHarness consistently improves visual fidelity and geometric accuracy across both text-only and multimodal coding agents. These findings highlight that modularizing visual reasoning into callable utilities provides a flexible and scalable foundation for future agentic coding systems.
BibTeX
@misc{yang2026vision,
  author = {Haolin Yang and Wenshan Wu and Jin Jiang and Yu Cheng and Yan Xia and Ting Song and Yangyu Huang and Zhiliang Peng and Wenhui Wang and Hangbo Bao and Yuzhe Zhang and Wen-Da Wei and Tengchao Lv and Li Dong and Lei Cui and Furu Wei},
  title = {{Vision-as-a-Tool: Visual Harness for Multimodal Agentic Coding}},
  year = {2026},
  eprint = {2611.17797},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17797},
}
Figure