WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models

Xinping Lei, Xinyu Che, Junqi Xiong, Chenchen Zhang, Yukai Huang, Chenyu Zhou, Haoyang Huang, Minghao Liu, Letian Zhu, Hongyi Ye, Jinhua Hao, Ken Deng, Zizheng Zhan, Han Li, Dailin Li, Yifan Yao, Ming Sun, Zhaoxiang Zhang, Jiaheng LiuarXiv 2026 · arXiv · OpenReview

TL;DR

Large language models (LLMs) are getting good at coding, but current benchmarks only test small parts of their abilities, like generating code from text. This paper introduces WebCompass, a comprehensive benchmark that evaluates LLMs on the full web development lifecycle (generation, editing, and repair) using multimodal inputs (text, image, video). WebCompass aims to provide a more realistic and holistic assessment of these AI coding agents.

What Problem Does This Paper Address?

The rapid advancement of LLMs has led to the development of AI agents capable of end-to-end web coding. However, existing benchmarks for evaluating these agents are limited. They often focus solely on text-based code generation and use static metrics that only check for functional correctness. This overlooks crucial aspects of real-world web development, such as:

  • Visual Fidelity: How good the generated website looks.
  • Interaction Quality: How well the website behaves when users interact with it.
  • Codebase-Level Reasoning: The ability to understand and modify entire codebases, not just isolated snippets.
  • Multimodality: The ability to understand and generate code based on various inputs like images and videos, not just text.

WebCompass aims to bridge this gap by providing a benchmark that covers the entire web development workflow, from initial creation to iterative refinement, and considers multiple input types and evaluation criteria that reflect real-world development challenges.

Methods

WebCompass introduces a novel evaluation framework with several key components:

  • Multimodal Inputs & Task Types: It supports text, image, and video inputs and covers three core task types: Generation (creating a website from scratch), Editing (modifying existing code based on instructions), and Repair (fixing bugs). This results in seven distinct task categories.
  • Human-in-the-Loop Data Curation: A rigorous pipeline involving human annotators was used to create high-quality benchmark instances, covering 15 generation domains, 16 editing operations, and 11 repair defect types, each categorized by difficulty (Easy/Medium/Hard).
  • Task-Aware Evaluation Paradigms:
    • For Editing and Repair, a checklist-guided LLM-as-a-Judge protocol is used. This involves an LLM evaluating generated code patches against specific criteria.
  • For Generation, a novel Agent-as-a-Judge paradigm is introduced. This involves an AI agent autonomously interacting with the generated website in a real browser, synthesizing test cases, and evaluating its functionality and visual appeal, mimicking human acceptance testing. This uses the Model Context Protocol (MCP) for browser control.

Datasets

The paper introduces WebCompass, a new multimodal benchmark. It is constructed through a multi-stage, human-in-the-loop pipeline. The dataset spans:

  • 15 Generation Domains: Covering areas like e-commerce, social media, productivity tools, etc.
  • 16 Editing Operation Types: Including tasks like data tables, rich text editors, drag-and-drop interfaces, and advanced animations.
  • 11 Repair Defect Types: Addressing visual layout issues, semantic correctness, and interactive usability problems.

The data is annotated with difficulty levels (Easy, Medium, Hard). WebCompass aims to be more comprehensive than existing benchmarks like SWE-bench, Interaction2Code, and DesignBench by covering generation, editing, and repair across multiple modalities.

Key Insights and Ideas

  • The Full Lifecycle Matters: Evaluating LLMs on the entire web development process (generation, editing, repair) reveals different strengths and weaknesses compared to single-task benchmarks.
  • Visuals are a Persistent Bottleneck: Even for top-performing models, achieving high visual fidelity and aesthetic quality remains a significant challenge, especially for open-source models.
  • Framework Choice Impacts Performance: The choice of front-end framework (React, Vue, Vanilla) can significantly influence model performance, with Vanilla often leading in generation and editing, while React can be beneficial for repair tasks.
  • Editing and Repair Stress Different Skills: Editing requires understanding and implementing new requirements, while repair focuses on precise defect localization and fixing. These tasks have distinct difficulty profiles.
  • Agent-as-a-Judge is Crucial for Generation: Evaluating code generation requires dynamic, interactive testing in a real browser to capture runtime behavior and user experience, which static tests cannot achieve.

Key Results

  • Closed-Source Models Lead: Closed-source models (like Claude-Opus-4.5 and Gemini-3-Pro-Preview) significantly outperform open-source models, achieving overall scores around 67% compared to the best open-source model's 41%.
  • Visual Quality is the Weakest Link: Across all models and tasks, visual dimensions (Design Quality, Style Conformance) consistently score lower than functional or execution metrics.
  • Framework Performance Varies: Vanilla (plain HTML/CSS/JS) generally leads in Generation and Editing, while Vue consistently underperforms.
  • Task-Type Difficulty: Editing tasks involving advanced animations are the hardest, while repair tasks involving semantic defects are also challenging.
  • Consistency is Key: Models that perform consistently across different subtasks and difficulty levels are favored by the harmonic mean scoring, highlighting the importance of robust performance over occasional peak scores.
  • Model Degradation with Complexity: As task difficulty increases (from Easy to Hard), model performance consistently decreases across all task categories and evaluation dimensions.

Limitations and Future Work

  • Front-end Focus: WebCompass currently only evaluates front-end development. Future work could extend the benchmark to include back-end capabilities (database design, server-side logic, API development) for a more complete full-stack assessment.
  • Structured Queries vs. Creative Intent: The benchmark prioritizes deterministic evaluation by refining underspecified queries into structured design documents. This means it primarily tests instruction-following rather than the ability to interpret vague, creative intent. Future work could explore benchmarks that explicitly measure creative divergence.
  • Limited Real-time Interaction with Dynamic Pages: The current evaluation protocols may struggle with highly dynamic web pages (e.g., browser-based games) that require real-time interaction and rapid state transitions. Accurately assessing such pages might require more advanced simulation capabilities.
  • Evaluation Cost: The Agent-as-a-Judge protocol, while thorough, is computationally expensive due to browser interaction and test case synthesis, potentially limiting its accessibility for some research groups.
  • Data Contamination Risk: As a static benchmark, WebCompass is susceptible to data contamination if future models are trained on its data. Periodic updates or dynamic task generation might be necessary.

Summary updated

Abstract

Large language models are rapidly evolving into interactive coding agents capable of end-to-end web coding, yet existing benchmarks evaluate only narrow slices of this capability, typically text-conditioned generation with static-correctness metrics, leaving visual fidelity, interaction quality, and codebase-level reasoning largely unmeasured. We introduce WebCompass, a multimodal benchmark that provides unified lifecycle evaluation of web engineering capability. Recognizing that real-world web coding is an iterative cycle of generation, editing, and repair, WebCompass spans three input modalities (text, image, video) and three task types (generation, editing, repair), yielding seven task categories that mirror professional workflows. Through a multi-stage, human-in-the-loop pipeline, we curate instances covering 15 generation domains, 16 editing operation types, and 11 repair defect types, each annotated at Easy/Medium/Hard levels. For evaluation, we adopt a checklist-guided LLM-as-a-Judge protocol for editing and repair, and propose a novel Agent-as-a-Judge paradigm for generation that autonomously executes generated websites in a real browser, explores interactive behaviors via the Model Context Protocol (MCP), and iteratively synthesizes targeted test cases, closely approximating human acceptance testing. We evaluate representative closed-source and open-source models and observe that: (1) closed-source models remain substantially stronger and more balanced; (2) editing and repair exhibit distinct difficulty profiles, with repair preserving interactivity better but remaining execution-challenging; (3) aesthetics is the most persistent bottleneck, especially for open-source models; and (4) framework choice materially affects outcomes, with Vue consistently challenging while React and Vanilla/HTML perform more strongly depending on task type.
BibTeX
@misc{lei2026webcompass,
  author = {Xinping Lei and Xinyu Che and Junqi Xiong and Chenchen Zhang and Yukai Huang and Chenyu Zhou and Haoyang Huang and Minghao Liu and Letian Zhu and Hongyi Ye and Jinhua Hao and Ken Deng and Zizheng Zhan and Han Li and Dailin Li and Yifan Yao and Ming Sun and Zhaoxiang Zhang and Jiaheng Liu},
  title = {{WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models}},
  year = {2026},
  eprint = {2604.18224},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.18224},
}
Figure