Abstract
MarkupLM is a novel pre-training framework designed for Visually Rich Document Understanding (VRDU) in documents with dynamic layouts, such as HTML and XML. By jointly training on text and the underlying Document Object Model (DOM) tree structure, it leverages structural hierarchy as a powerful proxy for visual layout. This approach allows MarkupLM to outperform existing models that rely on fixed 2D coordinates, proving that markup structure is a robust and efficient source of information for understanding web-based documents.
Introduction: The Challenge of Dynamic Layouts
In the field of Visually Rich Document Understanding (VRDU), traditional models like LayoutLM have achieved success by combining text with 2D layout information (x, y coordinates) and document images. However, these methods struggle with markup-language-based documents like HTML.
Unlike static files such as scanned PDFs, an HTML page renders differently depending on the environment. Because 2D positions are fluid, the authors argue that the DOM tree structure is the most reliable source of information for understanding how different pieces of text relate to one another. MarkupLM is designed to be "render-free," making it more lightweight and faster than models requiring heavy OCR or rendering engines.
Methodology: Architecture and Pre-training
MarkupLM builds upon the Transformer architecture but introduces several novel components to handle the specific "language" of markups.
XPath Embeddings
Instead of 2D coordinates, MarkupLM utilizes XPath, a query language that describes the path from the root of a document to a specific node. Each token's embedding includes its tag (e.g., div, span) and its subscript (the index of that tag among its siblings). These are processed through a dedicated XPath Extractor and a Feed-Forward Network (FFN), providing a coordinate-independent way to locate information.
Pre-training Objectives
The framework employs three novel tasks to align text with its structural context:
- Masked Markup Language Modeling (MMLM): Similar to standard masked language modeling, but the model must recover masked text tokens using surrounding markup clues.
- Node Relation Prediction (NRP): This task explicitly teaches the model the hierarchy of the DOM tree. The model is given pairs of nodes and must predict their relationship: are they siblings, parent and child, or ancestor and descendant?
- Title-Page Matching (TPM): A page-level task where the model predicts whether a specific
<title>string belongs to the<body>of the page, helping it learn high-level semantic summaries.
Experimental Results
MarkupLM achieved state-of-the-art results on major benchmarks using a pre-training corpus of 24 million English webpages from Common Crawl:
- WebSRC: On this web-based structural reading comprehension task, MarkupLM (Large) achieved an F1 score of 80.54, significantly outperforming previous multimodal models that used actual webpage images.
- SWDE: On the Structured Web Data Extraction task, the model reached an average F1 of 97.37. It demonstrated a massive lead in "few-shot" scenarios, where very little labeled data is available.
An ablation study revealed that the Title-Page Matching (TPM) task provided the most significant boost to performance (+4.6% EM), suggesting that global page context is vital for local understanding.
Conclusion
MarkupLM demonstrates that structural hierarchy is an effective proxy for vision in web documents. By encoding the XPath and DOM relationships, the model gains a robust understanding of document semantics that is independent of screen size or CSS styling. Future research directions include expanding the model to multilingual contexts and exploring "multi-view" pre-training that simultaneously utilizes both markup structure and rendered 2D layouts.
Summary updated