Document Level Partitioning for Robust Model Evaluation

19 concepts8 papers

When evaluating machine learning models on text or structured documents, the standard recipe is simple: take all your data points, shuffle them randomly, and split them into training and testing sets. While this approach works well for independent and identically distributed (i.i.d.) data, it breaks down catastrophically when applied to sub-components of larger documents—such as sentences, paragraphs, pages, or figures.

The core issue is data leakage. Documents possess an inherent narrative flow, a consistent vocabulary, specific entity distributions, and uniform stylistic templates. When we randomly shuffle individual instances into both training and testing pools, we inadvertently allow the model to cheat. Instead of learning to generalize to new, unseen concepts, the model memorizes the unique contextual signatures of the documents it encountered during training, artificially inflating its performance on the test set.

Mechanisms of Instance-Level Leakage

Data leakage manifests across various modalities and tasks whenever the boundaries of the parent document are ignored during dataset creation.

Entity and Vocabulary Overlap

In Named Entity Recognition (NER), standard random splitting frequently places identical entities into both training and testing sets. As demonstrated by Babl et al., this "entity leakage" allows models to simply memorize specific entity instances rather than learning generalizable linguistic patterns, masking the model's true inability to handle novel data.

Similarly, He 0002 et al. showed that sentences originating from the same source document share highly correlated vocabulary and context. Shuffling and splitting these sentences makes the test set artificially easy because the model has already seen the surrounding context during training.

Syntactic and Semantic Patterns

For tasks like protein-protein interaction (PPI) extraction, Airola et al. discovered that splitting datasets by individual entity pairs rather than by source documents exposes the model to identical sentence structures in both phases. The model learns to recognize specific sentence templates seen during training rather than understanding underlying semantic relationships. Even in tasks like metaphor detection, Brooks et al. noted that data handling and splitting strategies are highly critical, as the choice of text fragment boundaries directly influences model behavior and evaluation reliability.

Stylistic and Structural Consistency

Leakage is not confined to text; it also compromises structural tasks. In document layout analysis, consecutive pages of a report or manual share identical formatting templates. Pfitzmann et al. demonstrated that page-wise dataset splits allow models to exploit these repetitive templates, failing to reflect how the model would perform on entirely new document structures.

A similar phenomenon occurs in scientific figure captioning; Okazaki et al. noted that splitting by individual figures allows models to cheat by learning specific thematic or stylistic patterns shared among figures within the same paper.

Instance-Level Splitting (Flawed):
Document A ───► Sentence 1 ───► Train Set
            ───► Sentence 2 ───► Test Set (Leakage: Shared vocabulary/context)

Document-Level Splitting (Robust):
Document A ───► All Sentences ───► Train Set
Document B ───► All Sentences ───► Test Set  (Clean: Completely unseen narrative)

Why Document-Level Splitting Works

To build truly robust models, the evaluation framework must match the real-world deployment scenario, where a model encounters completely unfamiliar documents. The solution is document-level partitioning: ensuring that all instances originating from a single document are kept together and assigned exclusively to either the training, validation, or testing set.

By isolating entire documents, we eliminate the shared context that models exploit to bypass actual learning. For example, in predicting the citation worthiness of sentences, Gosangi et al. utilized document-level splits to guarantee that the model evaluates on the narrative flow of completely unseen papers. In specialized domains like archaeology, Brandsen et al. emphasized that robust document-level splitting strategies prevent ambiguous terms confined to specific documents from causing misleading evaluation metrics.

When document boundaries are strictly respected, the model can no longer rely on localized memorization. It is forced to learn higher-level, generalizable features—whether they are semantic structures, syntactic rules, or layout principles.

Quantifying the Inflation

How much does instance-level leakage actually distort our understanding of model performance? Across domains, the empirical evidence shows that random splitting creates a massive, artificial inflation of performance metrics:

  • Protein-Protein Interaction: Airola et al. found that pair-wise splitting instead of document-level splitting artificially inflates extraction metrics by up to 18 percentage points.
  • Document Layout Analysis: Pfitzmann et al. observed a roughly 10% overestimation in layout segmentation accuracy when using page-wise splits instead of document-aware splits.
  • Named Entity Recognition: Babl et al. quantified an inflation of 2–10% in NER performance scores due to entity leakage, prompting the authors to design graph-based "minimum cut" splitting methods to enforce clean boundaries.

These findings reveal that many state-of-the-art results may be artifacts of dataset design rather than architectural breakthroughs. Transitioning to document-level partitioning provides an honest benchmark, revealing the true generalization gaps that researchers must bridge.

Key Foundational Work

All-paths graph kernel for protein-protein interaction extraction with evaluation of cross-corpus learning (2008) — contributed 2 concept elements to this direction.
Metaphor Detection using Ensembles of Bidirectional Recurrent Neural Networks (2020) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_document_level_data_partitioning_0e43a796,
  title = {document-level data partitioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/document-level-data-partitioning/rdr_0e43a796ec1f47d597880b0aa4bddb22},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_document_level_data_partitioning_0e43a796_v1,
  title = {document-level data partitioning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/document-level-data-partitioning/rdr_0e43a796ec1f47d597880b0aa4bddb22/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake