Document Aware Data Splitting for Robust Evaluation

This direction is split into other directions: document-level data partitioning. Its last version stays readable and citable here.

Version 1 of 19 papers19 conceptsfrozen 2026-10-04

When training machine learning models on structured documents—whether they are scientific papers, legal contracts, or patents—it is tempting to treat every sentence, image, or page as an independent data point. However, across a wide range of tasks, researchers have discovered that this "instance-level" approach creates a subtle but devastating flaw: information leakage.

If a model sees one part of a document during training and another part of that same document during testing, it doesn't just learn the task; it learns the specific "fingerprint" of that document.

The Hidden Shortcut of Instance Splitting

The core problem lies in the consistency of human-authored documents. A single scientific paper typically uses a uniform vocabulary, a specific citation style, and a consistent visual layout.

In the domain of protein-protein interaction (PPI) extraction, Antti Airola, Sampo Pyysalo, and their colleagues identified that splitting datasets by individual entity pairs rather than by documents leads to a massive overestimation of performance. As detailed in All-paths graph kernel for protein-protein interaction extraction with evaluation of cross-corpus learning, if two different protein pairs appear in the same sentence, and one is placed in the training set while the other is in the test set, the model can "cheat." It recognizes the sentence structure it has already seen, allowing it to predict interactions based on memory rather than generalizable linguistic rules.

This isn't a minor statistical noise; Airola et al. found that this flaw can artificially inflate performance metrics by as much as 18 percentage points.

Memorizing Templates instead of Learning Features

The risk of leakage extends beyond text to visual and structural analysis. In document layout analysis, models are tasked with identifying headers, paragraphs, and figures. Birgit Pfitzmann and the team behind DocLayNet: A Large Human-Annotated Dataset for Document-Layout Segmentation demonstrated that page-wise splitting is equally dangerous.

Because a single document (like a financial report) uses the same margins, fonts, and header placements across all its pages, a model that trains on page 1 can easily "guess" the layout of page 2. By partitioning the dataset so that all pages of a document stay together in either the training or test set, the researchers found a 10% performance gap. This gap represents the difference between a model that has truly learned to understand document structures and one that has simply memorized a specific template.

Similarly, in the task of figure captioning, Naoaki Okazaki and his co-authors noted in SciCap+: A Knowledge Augmented Dataset to Study the Challenges of Scientific Figure Captioning that figures within the same paper often share recurring terminology and stylistic quirks. Document-level partitioning is the only way to ensure the model isn't just picking up on these document-specific themes.

Generalizing to the Narrative Flow

Moving to document-aware splitting changes what the model is forced to learn. Instead of recognizing local patterns, the model must grapple with the broader context.

When predicting "citation worthiness"—whether a sentence requires a reference—Rakesh Gosangi and his team at Bloomberg argued in On the Use of Context for Predicting Citation Worthiness of Sentences in Scholarly Articles that the model must understand the narrative flow of an entire paper. By splitting by document, they ensure the evaluation reflects the model's ability to handle completely unseen papers with different argumentative structures.

This methodology is also critical for specialized domains like chemical patents. Jiayuan He and the ChEMU 2020 researchers utilized stratified document-level splits to ensure that NLP models were learning the underlying logic of chemical synthesis rather than the idiosyncratic formatting of a particular patent office or author.

Why Document-Aware Splitting Works

The intuition behind this research direction is simple: The unit of independence in the data should match the unit of independence in the real world.

In practice, we don't need a model to tell us about a sentence in a document it has already partially read; we need it to work on a brand-new document it has never seen before. By enforcing document-level boundaries during the data split:

  1. We eliminate "contextual leakage": The model cannot use nearby sentences or pages as a crutch.
  2. We prevent "template memorization": The model must learn general features (like "what does a header look like?") rather than specific ones (like "this specific document uses 14pt Bold Arial for headers").
  3. We obtain "honest" metrics: The resulting accuracy scores reflect true generalization, providing a reliable estimate of how the model will perform in production environments.

By shifting the focus from individual instances to the source documents, researchers like Airola, Pfitzmann, and Gosangi have established a more rigorous standard for evaluation in the age of large-scale document AI.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_document_aware_data_splitting_for_robust_0e6a8c14,
  title = {Document Aware Data Splitting for Robust Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/document-aware-data-splitting-for-robust-evaluation/rdr_0e6a8c14bb8a4b68ac27dd7468b501ae},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_document_aware_data_splitting_for_robust_0e6a8c14_v1,
  title = {Document Aware Data Splitting for Robust Evaluation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/document-aware-data-splitting-for-robust-evaluation/rdr_0e6a8c14bb8a4b68ac27dd7468b501ae/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}