Unsupervised Exemplar-Based Learning for Improved Document Image Classification

Sherif Abuelwafa, Marco Pedersoli, Mohamed CherietIEEE Access 2019 · OpenReview

Abstract: Many recent state-of-the-art approaches for document image classification are based on supervised feature learning that requires a large amount of labeled training data. In real-world problem of document image classification, the available amount of labeled data is limited and scarce while a large amount of unlabeled data is often available at almost no cost. In this paper, we present an approach for learning visual features for document analysis in an unsupervised way, which improves the document image classification performance without increasing the amount of annotated data. The proposed approach trains a neural network model on an auxiliary task in which every training example is associated with a different label (exemplar) and expanded to multiple images through a data augmentation technique. Thus, the learned model, which is trained in an unsupervised way, is used to boost the document classification performance. In fact, this learned model has proved to be consistently efficient in two different settings: i) as an unsupervised feature extractor to represent document images for an unsupervised classification task (i.e., clustering); and ii) in the parameters initialization of a supervised classification task trained with a small amount of annotated data. We perform experiments on the Tobacco-3482 dataset and demonstrate the capability of our approach to improve i) the unsupervised classification accuracy up to 2.4%; and ii) the supervised classification accuracy by 1.5% without any extra data or by 5% when using 3000 additional not annotated samples.

Introduction

Deep learning has revolutionized document image classification, enabling systems to distinguish between forms, resumes, and letters with high precision. However, these models typically rely on massive labeled datasets like RVL-CDIP. In practical scenarios, labeling thousands of documents is a bottleneck—it is slow, expensive, and often requires domain-specific knowledge.

While unlabeled data is abundant, standard unsupervised methods often fail to capture the specific spatial and structural nuances of document layouts. This research addresses the gap by leveraging "exemplar-based learning," an unsupervised technique that treats each unlabeled image as its own unique category to learn discriminative structural features.

Proposed Methodology: Exemplar-Based Learning

The core of the proposed approach is a two-stage process: unsupervised pre-training followed by downstream classification.

Unsupervised Pre-training

The researchers take $N$ unlabeled images. Each image is considered a "surrogate class." A Convolutional Neural Network (CNN), based on the AlexNet architecture, is trained on an auxiliary task: identifying which "seed" image a specific variation belongs to. This forces the network to learn features that are invariant to transformations, essentially capturing the core visual "DNA" of the document layout.

Overview of the unsupervised exemplar-based pre-training pipeline.

Data Augmentation

To create the surrogate classes, heavy data augmentation is applied to each seed image. Transformations include rotations of $\pm90^\circ$, zooming, and horizontal flips. By training the model to recognize these variations as belonging to the same original exemplar, the model learns to prioritize global structural patterns over local pixel noise.

Example of data augmentation applied to a single document image (exemplar) to create a surrogate class.

Downstream Applications

The pre-trained model is utilized in two primary ways:

  1. Unsupervised Clustering: The model acts as a feature extractor. Document images are passed through the network, and the resulting feature vectors are clustered using K-means.
  2. Supervised Fine-tuning: The pre-trained weights serve as a "warm start" for a standard supervised task. Instead of random initialization, the model begins with a sophisticated understanding of document layouts, which is then refined using a small set of labeled data.

Experimental Results

The approach was evaluated primarily on the Tobacco-3482 dataset (3,482 images across 10 classes). To test scalability, the authors also used subsets of the RVL-CDIP dataset (400,000 images) as additional unlabeled data.

  • Unsupervised Performance: Using learned features for clustering improved accuracy by 2.4% compared to previous state-of-the-art unsupervised methods.
  • Supervised Performance:
    • Training on Tobacco-3482 from scratch yielded a median accuracy of 63.38%.
  • By pre-training on the same data (unlabeled), accuracy rose to 65.01%.
  • By adding 3,000 unlabeled images from RVL-CDIP during pre-training, accuracy jumped to 68.86%—a total improvement of 5.48% without adding a single extra human label.

Key Insights

A significant finding was that features from the flatten layer (immediately following convolutions) outperformed those from deeper dense layers. This suggests that the flatten layer better preserves the spatial locality of features—a critical factor in document layout analysis—whereas fully connected layers tend to lose this spatial context.

Conclusion

Exemplar-based learning provides a powerful framework for utilizing unlabeled document data. By treating individual images as classes, CNNs can learn robust, layout-aware features that significantly enhance both unsupervised clustering and supervised classification, particularly when labeled data is scarce. Future work may explore modern architectures like Vision Transformers and the integration of OCR-based textual features to further improve performance.

Summary updated

Abstract

Many recent state-of-the-art approaches for document image classification are based on supervised feature learning that requires a large amount of labeled training data. In real-world problem of document image classification, the available amount of labeled data is limited and scarce while a large amount of unlabeled data is often available at almost no cost. In this paper, we present an approach for learning visual features for document analysis in an unsupervised way, which improves the document image classification performance without increasing the amount of annotated data. The proposed approach trains a neural network model on an auxiliary task in which every training example is associated with a different label (exemplar) and expanded to multiple images through a data augmentation technique. Thus, the learned model, which is trained in an unsupervised way, is used to boost the document classification performance. In fact, this learned model has proved to be consistently efficient in two different settings: i) as an unsupervised feature extractor to represent document images for an unsupervised classification task (i.e., clustering); and ii) in the parameters initialization of a supervised classification task trained with a small amount of annotated data. We perform experiments on the Tobacco-3482 dataset and demonstrate the capability of our approach to improve i) the unsupervised classification accuracy up to 2.4%; and ii) the supervised classification accuracy by 1.5% without any extra data or by 5% when using 3000 additional not annotated samples.
BibTeX
@inproceedings{abuelwafa2019unsupervised,
  author = {Sherif Abuelwafa and Marco Pedersoli and Mohamed Cheriet},
  title = {{Unsupervised Exemplar-Based Learning for Improved Document Image Classification}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=-k-txISrBv},
}
Figure