TL;DR
AmbiHTMLBench is a new benchmark designed to push document parsing beyond simple text extraction by requiring models to generate full, executable HTML/CSS from document images. It introduces a dual-evaluation framework—combining automated objective metrics with human-aligned LLM judges—to reveal that current state-of-the-art models often struggle to balance textual accuracy with visual and structural fidelity.
What Problem Does This Paper Address?
Most current document parsing systems rely on Markdown as their primary output. While Markdown is excellent for linear text, it fails to capture the "visually rich" nature of complex documents—think multi-column magazine layouts, intricate table geometries, and hierarchical headers. For AI agents to truly interact with documents (e.g., "click the button in the third row" or "edit the caption of the second figure"), they need a representation that is addressable, editable, and renderable.
The authors argue that HTML is the ideal "bridge" for this task. However, existing benchmarks either focus on isolated components (like OCR or table recognition) or target simple webpage screenshots. There hasn't been a systematic way to evaluate how well a model can reconstruct a heterogeneous, real-world document into a functional HTML artifact that preserves content, structure, and appearance simultaneously.
Methods
The authors define a Document-to-HTML task where the input is a page image and the output is executable HTML/CSS. To evaluate this, they propose two complementary protocols:
- Objective Evaluation: The predicted HTML is rendered in a headless browser. The system then extracts visible text blocks and tables, aligning them with ground-truth annotations using a thresholded Hungarian matching algorithm. It measures three dimensions:
- Content Fidelity: Page-level text edit distance and Table Edit Distance (TEDS).
- Structural Fidelity: Hierarchy and block relationships.
- Layout Consistency: A novel 2D layout graph similarity score that checks if relative positions and sizes are preserved.
- Subjective Evaluation: Recognizing that automated metrics don't always capture "human-perceived" quality, the researchers built a Human-calibrated Proxy. They collected nearly 8,000 human pairwise judgments and used them to train a committee of five VLM judges (including GPT and Claude variants). This "frozen proxy" allows for scalable, human-aligned evaluation of visual fidelity.
Datasets
The core contribution is AmbiHTMLBench, a dataset of 1,115 diverse pages with expert-level structured annotations.
- Diversity: The pages span four "layout families": Institutional PDFs (reports, datasheets), Web-native pages, Newspapers/Magazines, and Posters/Designed pages.
- Construction: The authors started with a pool of over 50,000 candidates, using layout-aware clustering (via ResNet-50 features) and iterative filtering to ensure a wide variety of styles and difficulties while removing near-duplicates.
- Statistics: The final set averages 2,329 characters and 26 blocks per page, with over half the pages containing more than 1,000 characters.
Key Insights and Ideas
- HTML is Actionable: Pilot studies (shown in Figure 3) demonstrate that agents using HTML representations achieve significantly higher accuracy in tasks like field-to-slot binding (62% vs 44%) compared to non-HTML records, because the DOM makes elements explicitly addressable.
- The Fidelity Trade-off: A major "aha moment" was discovering that the "objective" winner and "subjective" winner are often different. A model might get every word right (high objective score) but break the column layout, making the document unreadable to a human. Conversely, a model might hallucinate minor text but produce a visually perfect reconstruction that humans prefer.
- Task-Specific Adaptation: The researchers found that even small-parameter models (like Qwen 4B/9B) can be significantly improved through fine-tuning on high-quality document-to-HTML data, sometimes outperforming much larger unadapted models.
Key Results
The benchmark evaluated 15 primary multimodal systems.
- Objective Leaderboard: Gemini 3.1 Pro Preview took the top spot with an objective score of 0.8092, leading in text accuracy and table fidelity.
- Subjective Leaderboard: The results shifted here, with GPT-5.6-Sol emerging as the winner. This suggests that while Gemini is more "accurate" on a character level, the GPT model produces layouts that humans find more faithful to the original visual intent.
- Fine-tuning Gains: Using LoRA-based supervised fine-tuning (SFT), the authors improved a Qwen 9B model's weighted score from 0.6623 to 0.7064, proving the benchmark's utility as a training target.
Limitations and Future Work
The authors acknowledge several areas for growth:
- Single-Page Focus: The current benchmark evaluates documents page-by-page. Future iterations could look at multi-page consistency and cross-page element linking.
- Proxy Asymmetry: The subjective evaluation proxy, while highly correlated with humans, has some inherent biases based on the specific VLM judges used in the committee.
- Dynamic Content: The benchmark focuses on static document reconstruction; extending this to interactive elements (like functional forms or embedded videos) remains a challenge for future research.
Summary updated