Abstract
Multimodal pre-training with text, layout, and image has made significant progress for Visually Rich Document Understanding (VRDU), especially for fixed-layout documents such as scanned document images. However, there are still a large number of digital documents where the layout information is not fixed and needs to be interactively and dynamically rendered for visualization, making existing layout-based pre-training approaches difficult to apply. In this paper, we propose MarkupLM for document understanding tasks with markup languages as the backbone, such as HTML/XML-based documents, where text and markup information is jointly pre-trained. Experiment results show that the pre-trained MarkupLM significantly outperforms the existing strong baseline models on several document understanding tasks.
Introduction: The Challenge of Dynamic Layouts
Most recent breakthroughs in Visually Rich Document Understanding (VRDU)—like the LayoutLM family—rely heavily on 2D layout information (bounding boxes) and document images. This works effectively for scanned receipts or PDFs where the text stays exactly where it was printed.
However, a massive portion of the world's data lives in markup languages (HTML/XML). These documents are dynamic: the layout changes depending on whether you are viewing it on a phone, a tablet, or a desktop, as shown in Figure 1. Because 2D coordinates are not "fixed" in the source code, traditional layout-based models struggle. The authors argue that for these documents, the DOM (Document Object Model) tree structure is a more reliable "anchor" for understanding than visual coordinates.
Methods: Leveraging the DOM Tree
MarkupLM adopts a Transformer-based architecture (similar to BERT or RoBERTa) but introduces several innovations to ingest markup structure without the need for rendering:
- XPath Embedding: This is the core novelty. For every token, the model calculates its XPath (e.g.,
/html/body/div[1]/span[2]). It breaks this path into tag-subscript pairs, embeds them, and aggregates them to give the model a sense of exactly where that token sits in the document hierarchy. This process is illustrated in Figure 2. - New Pre-training Objectives:
- Masked Markup Language Modeling (MMLM): Similar to standard MLM, but the model must recover masked tokens using both text and markup clues.
- Node Relation Prediction (NRP): The model is given pairs of nodes and must predict their relationship (e.g., Is node A the parent, child, or sibling of node B?).
- Title-Page Matching (TPM): A page-level task where the model predicts if a specific
<title>belongs to a given<body>.
Crucially, MarkupLM is render-free. It doesn't need to actually "see" the webpage or calculate bounding boxes, making it much faster and more lightweight than models that require OCR or image features.
Experimental Evaluation
The authors pre-trained the model on a subset of the Common Crawl (CC) dataset containing 24 million English webpages. They evaluated the model on two major benchmarks:
- WebSRC: A Web-based Structural Reading Comprehension dataset (440K QA pairs).
- SWDE: The Structured Web Data Extraction dataset, which involves pulling specific attributes (like "Price" or "Author") from websites across different verticals (Auto, Book, Camera, etc.).
Key Results and Insights
MarkupLM significantly outperformed strong baselines (like BERT and RoBERTa) and specialized web-extraction models:
- WebSRC: On the "Large" version of the model, MarkupLM achieved an Exact Match (EM) score of 74.43%, a significant improvement over the H-PLM (Text + HTML) baseline of 69.57%.
- SWDE: In few-shot settings (using only a few "seed" sites for training), MarkupLM outperformed the previous state-of-the-art, SimpDOM, by a wide margin. For example, with only one seed site ($k=1$), MarkupLM-Large achieved an F1 of 85.71% compared to SimpDOM's 83.06%.
Structure over Coordinates: For digital-native documents, the logical hierarchy (the tree) is often more informative than the visual arrangement. XPath provides a robust coordinate system for this hierarchy. Furthermore, by using embeddings rather than raw HTML tags, MarkupLM avoids bloating sequence lengths, allowing it to process longer documents efficiently.
Conclusion and Future Work
MarkupLM demonstrates that pre-training on markup structure is a powerful alternative to layout-based methods for dynamic digital documents. While the current model focuses on English HTML, future work includes expanding to multilingual support and other XML-based formats like digital-born PDFs or Office documents. Additionally, future iterations might explore hybrid approaches that combine XPath with lightweight visual cues for elements like icons or specific CSS styling.
Summary updated