While modern document image classifiers achieve near-perfect scores on standard benchmarks, they are surprisingly brittle when deployed in real-world environments. A new study highlights this vulnerability by introducing a benchmark for Out-of-Distribution (OOD) detection, revealing that state-of-the-art models suffer accuracy drops of 15% to 30% when processing modern, "born-digital" documents or unseen categories.
The "Robustness Gap" in Document Classification
The current "gold standard" for training and evaluating document image classifiers is the RVL-CDIP dataset. However, RVL-CDIP acts as a time capsule: it consists almost entirely of grayscale, scanned documents from the American tobacco industry created before 2006.
In production, models inevitably encounter "distribution shifts." These can include modern digital PDFs, documents from different industries, or entirely new document types like music sheets or court records. Because current research heavily over-optimizes for the narrow RVL-CDIP distribution, it creates a false sense of security regarding how these models will perform in the wild.
A New Benchmark for OOD Detection
To measure this robustness gap, researchers curated a new OOD benchmark sourced from web searches and DocumentCloud, split into two distinct challenges:
- RVL-CDIP-N (Near-OOD): Contains 1,002 images that fit into the original 16 RVL-CDIP categories (e.g., "email," "invoice," "memo") but originate from different, modern sources. These are mostly "born-digital" documents from the 2010s and 2020s, lacking the scanner noise and specific aesthetic of the original training set.
- RVL-CDIP-O (Far-OOD): Contains 3,415 images of document types not present in the original 16 categories, such as maps and court documents.
Methodology
The study evaluated a suite of popular architectures to see how they handled these distribution shifts. Tested models included Convolutional Neural Networks (VGG-16, ResNet-50, GoogLeNet, AlexNet) and Transformers (LayoutLMv2 and the image-only DiT).
To measure OOD performance—essentially testing if a model "knows what it doesn't know"—the researchers used two primary decision functions: Maximum Softmax Probability (MSP) and the Energy Score, a technique utilizing raw output logits that often outperforms MSP in anomaly detection.
Key Insights and Results
The evaluation revealed several critical vulnerabilities in current document classification models:
- The "Clean Data" Paradox: Surprisingly, models trained on noisy, old scans perform worse on clean, modern digital documents. While one might expect a model to find pristine text easier to read, the shift in visual style alone is enough to confuse them.
- Generalization vs. Specialization: Even the best-performing model, the Document Image Transformer (DiT), saw its accuracy plummet from ~93% on the standard test set to ~78% on the Near-OOD set. This suggests models are memorizing the specific "look" of 1990s tobacco memos rather than learning the abstract concept of a memo.
- Accuracy Degradation: Across the board, all models showed a substantial performance hit on RVL-CDIP-N, with accuracy drops ranging from 15% to 30% compared to the original benchmark.
- The Difficulty of "None of the Above": Models struggle significantly to distinguish between a shifted in-domain document (Near-OOD) and a completely unknown document type (Far-OOD). When using the Energy score to detect Far-OOD documents, DiT achieved an Area Under the Curve (AUC) of 0.888. However, when tasked with distinguishing between Near-OOD and Far-OOD, the AUC dropped to 0.753. This is a major hurdle for autonomous systems that need to flag unknown inputs for human review.
Limitations and Future Directions
While this new benchmark is a crucial step forward, it is relatively small (approx. 4,400 images) compared to the 400,000 images in the original RVL-CDIP dataset. Furthermore, for multimodal models like LayoutLMv2, performance is heavily tied to the quality of the underlying OCR engine (Tesseract was used in this study).
The authors conclude that future research must pivot away from merely chasing higher accuracy on the static RVL-CDIP leaderboard. Instead, the focus should shift toward training techniques that explicitly improve OOD robustness, such as advanced data augmentation that mimics various document eras and styles.
Summary updated