AmbiHTMLBench: A Benchmark for HTML-based Visually Rich Document parsing

Boyu Niu, Yingjie Xiong, Shurui Li, Linhan Pang, Yuntong Zhang, Han Wang, Hao Wu, Kai Zuo, Xu Tang, Xuanhe Zhou, Fan WuarXiv 2026 · arXiv

TL;DR

AmbiHTMLBench is a new benchmark designed to push document parsing beyond simple text extraction by requiring models to generate full, executable HTML/CSS from document images. It introduces a dual-evaluation framework—combining automated objective metrics with human-aligned LLM judges—to reveal that current state-of-the-art models often struggle to balance textual accuracy with visual and structural fidelity.

What Problem Does This Paper Address?

Most current document parsing systems rely on Markdown as their primary output. While Markdown is excellent for linear text, it fails to capture the "visually rich" nature of complex documents—think multi-column magazine layouts, intricate table geometries, and hierarchical headers. For AI agents to truly interact with documents (e.g., "click the button in the third row" or "edit the caption of the second figure"), they need a representation that is addressable, editable, and renderable.

The authors argue that HTML is the ideal "bridge" for this task. However, existing benchmarks either focus on isolated components (like OCR or table recognition) or target simple webpage screenshots. There hasn't been a systematic way to evaluate how well a model can reconstruct a heterogeneous, real-world document into a functional HTML artifact that preserves content, structure, and appearance simultaneously.

The pipeline from document images to reusable HTML across various genres.

Methods

The authors define a Document-to-HTML task where the input is a page image and the output is executable HTML/CSS. To evaluate this, they propose two complementary protocols:

  1. Objective Evaluation: The predicted HTML is rendered in a headless browser. The system then extracts visible text blocks and tables, aligning them with ground-truth annotations using a thresholded Hungarian matching algorithm. It measures three dimensions:
  2. Content Fidelity: Page-level text edit distance and Table Edit Distance (TEDS).
  3. Structural Fidelity: Hierarchy and block relationships.
  4. Layout Consistency: A novel 2D layout graph similarity score that checks if relative positions and sizes are preserved.
  5. Subjective Evaluation: Recognizing that automated metrics don't always capture "human-perceived" quality, the researchers built a Human-calibrated Proxy. They collected nearly 8,000 human pairwise judgments and used them to train a committee of five VLM judges (including GPT and Claude variants). This "frozen proxy" allows for scalable, human-aligned evaluation of visual fidelity.
The three levels of fidelity (content, structural, visual) measured by the benchmark.

Datasets

The core contribution is AmbiHTMLBench, a dataset of 1,115 diverse pages with expert-level structured annotations.

  • Diversity: The pages span four "layout families": Institutional PDFs (reports, datasheets), Web-native pages, Newspapers/Magazines, and Posters/Designed pages.
  • Construction: The authors started with a pool of over 50,000 candidates, using layout-aware clustering (via ResNet-50 features) and iterative filtering to ensure a wide variety of styles and difficulties while removing near-duplicates.
  • Statistics: The final set averages 2,329 characters and 26 blocks per page, with over half the pages containing more than 1,000 characters.

Key Insights and Ideas

  • HTML is Actionable: Pilot studies (shown in Figure 3) demonstrate that agents using HTML representations achieve significantly higher accuracy in tasks like field-to-slot binding (62% vs 44%) compared to non-HTML records, because the DOM makes elements explicitly addressable.
  • The Fidelity Trade-off: A major "aha moment" was discovering that the "objective" winner and "subjective" winner are often different. A model might get every word right (high objective score) but break the column layout, making the document unreadable to a human. Conversely, a model might hallucinate minor text but produce a visually perfect reconstruction that humans prefer.
  • Task-Specific Adaptation: The researchers found that even small-parameter models (like Qwen 4B/9B) can be significantly improved through fine-tuning on high-quality document-to-HTML data, sometimes outperforming much larger unadapted models.

Key Results

The benchmark evaluated 15 primary multimodal systems.

  • Objective Leaderboard: Gemini 3.1 Pro Preview took the top spot with an objective score of 0.8092, leading in text accuracy and table fidelity.
  • Subjective Leaderboard: The results shifted here, with GPT-5.6-Sol emerging as the winner. This suggests that while Gemini is more "accurate" on a character level, the GPT model produces layouts that humans find more faithful to the original visual intent.
  • Fine-tuning Gains: Using LoRA-based supervised fine-tuning (SFT), the authors improved a Qwen 9B model's weighted score from 0.6623 to 0.7064, proving the benchmark's utility as a training target.
The objective leaderboard showing performance across text, table, and layout metrics.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Single-Page Focus: The current benchmark evaluates documents page-by-page. Future iterations could look at multi-page consistency and cross-page element linking.
  • Proxy Asymmetry: The subjective evaluation proxy, while highly correlated with humans, has some inherent biases based on the specific VLM judges used in the committee.
  • Dynamic Content: The benchmark focuses on static document reconstruction; extending this to interactive elements (like functional forms or embedded videos) remains a challenge for future research.

Summary updated

Abstract

Traditional document parsing systems use Markdown as their output representation, which is effective for linear text but does not fully preserve fine-grained spatial structure, table geometry, visual hierarchy, or rendering semantics. We show that HTML is advantageous for heterogeneous and layout-sensitive documents whose representations must remain editable, inspectable, and reusable, and introduce AmbiHTMLBench to systematically study this setting. In this benchmark, given a real-world document, a model must recover executable HTML that preserves its content, organization, and visual appearance. The benchmark contains 1,115 diverse pages with structured annotations and evaluates predictions in two different ways: The objective protocol aligns ground-truth and predicted rendered blocks and measures table fidelity, page-level text error, and a two-dimensional layout consistency; a complementary reference-anchored protocol measures perceived fidelity by aligning LLM judges with human preferences. Across ten primary multimodal systems, the best objective score is 0.8092, while the subjective winner is different, demonstrating that automatic decomposition and human-perceived fidelity capture complementary properties. We further fine-tune a small-parameter expert model and find that task-specific adaptation yields good performance on document-to-HTML reconstruction task. These results position AmbiHTMLBench as a diagnostic testbed for building smarter and more logical document parsing systems.
BibTeX
@misc{niu2026ambihtmlbench,
  author = {Boyu Niu and Yingjie Xiong and Shurui Li and Linhan Pang and Yuntong Zhang and Han Wang and Hao Wu and Kai Zuo and Xu Tang and Xuanhe Zhou and Fan Wu},
  title = {{AmbiHTMLBench: A Benchmark for HTML-based Visually Rich Document parsing}},
  year = {2026},
  eprint = {2611.06449},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06449},
}
Figure