WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics

Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang, Tao Xie

TL;DR

Large language models (LLMs) are getting good at generating web applications, but evaluating them is tough. This paper introduces WebCoderBench, a new benchmark with real-world user requirements and 24 comprehensive, interpretable metrics. It reveals that while LLMs are improving rapidly, no single model dominates, offering developers clear targets for optimization.

What Problem Does This Paper Address?

The rise of LLMs has opened up exciting possibilities for automatically generating web applications from user requirements. However, existing benchmarks for evaluating these LLM-generated web apps fall short in several key areas:

  • Authenticity: Many benchmarks use synthetic or expert-curated requirements, failing to capture the diverse and sometimes ambiguous ways real users express their needs.
  • Generality: Evaluating LLM outputs often relies on fixed ground-truth implementations or test cases, which isn't practical for the open-ended nature of web app generation where multiple valid solutions exist.
  • Interpretability: Existing evaluations often provide a single score, making it hard to understand why a model performed well or poorly, and how to improve it. User satisfaction is multi-dimensional, and current metrics don't reflect this.

Methods

To tackle these challenges, the authors developed WebCoderBench, a novel benchmark with a multi-faceted approach:

  • Dataset Collection: They gathered 1,572 real-world user requirements from an industrial partner, spanning diverse modalities (text, images, URLs) and expression styles (technical to colloquial). This dataset aims to be authentic and representative of actual user needs.
  • Comprehensive Evaluation Metrics: WebCoderBench introduces 24 fine-grained evaluation metrics across 9 perspectives. These metrics combine:
  • Rule-based metrics: Leveraging established standards (like Google Lighthouse, Lint) and internal industry guidelines.
  • LLM-as-a-judge paradigm: Using LLMs to assess aspects like functional correctness, visual alignment, and content alignment, referencing human-annotated ground-truth checklists.
  • Human-Preference-Aligned Weighting: To ensure interpretability and fairness, they conducted a survey to gather human preferences on the importance of different metrics. These preferences were used to assign weights, creating an overall score that reflects real-world user priorities.
  • Z-score Normalization: To compare metrics on a comparable scale and emphasize the impact of weights, they used z-scores for calculating the overall model performance.

Datasets

  • WebCoderBench Dataset: The core of this work is the introduction of the WebCoderBench dataset, comprising 1,572 real-world user requirements. This dataset is unique in its collection from actual user interactions and its diversity in modalities, clarity, and complexity.
  • Existing Benchmarks: The paper contrasts WebCoderBench with existing benchmarks, categorizing them into synthetic, reference-based, and arena-style, highlighting their limitations in authenticity, generality, and interpretability.

Key Insights and Ideas

  • No Single Dominant Model: The evaluation revealed that no single LLM or agent consistently outperforms others across all metrics. This is a crucial insight for developers, showing that optimization needs to be targeted.
  • Rapid Evolution of LLMs: Newer models, especially those released within the past year, consistently score higher, demonstrating the accelerated pace of development in the LLM space. Older models like GPT-40 are falling behind significantly.
  • Open-Source vs. Closed-Source Gap Narrowing: While closed-source models like GPT-5-High still lead, the performance of leading open-source models (GLM-4.5, MiniMax-M2) is rapidly closing the gap, indicating strong progress in the open-source community.
  • Agents vs. Base Models: LLM-based agents generally perform well but sometimes fall short of their base models, suggesting that while agents offer enhanced capabilities (like online resource access), they might come at the cost of performance or visual consistency.
  • User Preferences Drive Evaluation: The paper emphasizes the importance of aligning evaluation metrics with human preferences, showing how different user personas prioritize different aspects of web app quality.

Key Results

  • GPT-5-High achieved the highest overall score, demonstrating strong performance across most perspectives.
  • GLM-4.5 was the top-performing open-source model, showing balanced capabilities across nine perspectives.
  • MiniMax-M2 also performed competitively among open-source models.
  • GPT-40 showed the weakest effectiveness, with scores consistently below average.
  • The weighted z-scores revealed that even small differences in scores can be meaningful due to the weighting scheme.
  • Analysis of question types showed that models perform better on vague requirements and colloquial expression styles, while image-based inputs can degrade effectiveness.

Limitations and Future Work

  • Web Front-End Focus: The current benchmark focuses solely on web front-end applications and native HTML. Future work aims to incorporate backend tasks and support common frameworks like React.
  • Data Privacy: The dataset and evaluation metrics cannot be released publicly due to internal legal constraints related to user data. The authors plan to prepare another batch for open-sourcing.
  • Manual Annotation Reliability: While efforts were made to mitigate this (e.g., two-stage labeling, triple annotation for critical steps), manual annotation inherently has subjectivity.
  • Metric Design: The authors acknowledge that the metric suite can be further enriched and plan to expand it in future work.
  • LLM-as-a-Judge Stability: The paper mentions using the Mann-Whitney U test to verify score variance for LLM-as-a-judge metrics, but further work on metric stability and robustness is always valuable.
  • Future Work: The authors plan to incorporate backend tasks, support more frameworks, and prepare a new dataset for open-sourcing to further validate their findings. They also aim to enrich the metric set.

Summary updated

Abstract

Web applications (web apps) have become a key arena for large language models (LLMs) to demonstrate their code generation capabilities and commercial potential. However, building a benchmark for LLM-generated web apps remains challenging due to the need for real-world user requirements, generalizable evaluation metrics without relying on ground-truth implementations or test cases, and interpretable evaluation results. To address these challenges, we introduce WebCoderBench, the first real-world-collected, generalizable, and interpretable benchmark for web app generation. WebCoderBench comprises 1,572 real user requirements, covering diverse modalities and expression styles that reflect realistic user intentions. WebCoderBench provides 24 fine-grained evaluation metrics across 9 perspectives, combining rule-based and LLM-as-a-judge paradigm for fully automated, objective, and general evaluation. Moreover, WebCoderBench adopts human-preference-aligned weights over metrics to yield interpretable overall scores. Experiments across 12 representative LLMs and 2 LLM-based agents show that there exists no dominant model across all evaluation metrics, offering an opportunity for LLM developers to optimize their models in a targeted manner for a more powerful version.
BibTeX
@inproceedings{liu2026webcoderbench,
  author = {Chenxu Liu and Yingjie Fu and Wei Yang and Ying Zhang and Tao Xie},
  title = {{WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metrics}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.535},
}
Figure