Learning Surrogate Models of Document Image Quality Metrics for Automated Document Image Processing

Prashant Singh, Ekta Vats, Anders Hast

Abstract

Computation of document image quality metrics often depends upon the availability of a ground truth image corresponding to the document. This limits the applicability of quality metrics in applications such as hyperparameter optimization of image processing algorithms that operate on-the-fly on unseen documents. This work proposes the use of surrogate models to learn the behavior of a given document quality metric on existing datasets where ground truth images are available. The trained surrogate model can later be used to predict the metric value on previously unseen document images without requiring access to ground truth images. The surrogate model is empirically evaluated on the Document Image Binarization Competition (DIBCO) and the Handwritten Document Image Binarization Competition (H-DIBCO) datasets.

Introduction

In the field of document digitization, binarization—the conversion of gray-scale or color scans into clean black-and-white images—is a fundamental preprocessing step. The performance of binarization algorithms is highly dependent on hyperparameter tuning. Traditionally, this tuning is performed by comparing the algorithm's output against a manually curated "ground truth" image using metrics such as F-Measure or Peak Signal-to-Noise Ratio (PSNR).

However, a significant challenge arises in real-world production environments where ground truth images for new documents are unavailable. This creates a circular dependency: one cannot calculate the quality metric without a reference, yet the metric is required to automatically optimize the processing algorithm. This research addresses this problem by developing surrogate models that can estimate these quality metrics without requiring reference images.

Methodology

The proposed approach treats the quality metric as a function to be approximated through surrogate modeling (or metamodeling). This involves training a computationally efficient mathematical model to mimic the behavior of a complex metric like the F-Measure.

Training Pipeline

The training process utilizes existing datasets where ground truth is available. As shown in Figure 1, the system computes relative image quality metrics (I) by comparing the original degraded images (X) with their processed versions (P). These relative metrics—such as PSNR, Distance Reciprocal Distortion (DRD), and Negative Rate Metric (NRM)—serve as features for the surrogate model. The target for training is the actual F-Measure calculated using the ground truth images (G).

Figure 1: The training phase of the surrogate model, illustrating how relative metrics and ground truth are used to train the model.

Model Architectures

Three distinct regression architectures were evaluated:

  1. Support Vector Regression (SVR): Utilizing a Sequential Minimal Optimization (SMO) solver.
  2. Artificial Neural Networks (ANN): Specifically, feed-forward back-propagation networks.
  3. Gaussian Processes (GP): Stochastic models that provide both predictions and uncertainty measures.

Experimental Setup and Results

The study utilized 86 images from the DIBCO (2009, 2011, 2013) and H-DIBCO (2010, 2012, 2014, 2016) datasets. These documents contain various historical degradations, including ink bleed-through, stains, and low contrast. 63 images were used for training, while 23 were reserved for testing.

Performance Analysis

Support Vector Regression (SVR) emerged as the most robust model, achieving the lowest Root Relative Square Error (RRSE) of 0.8053. A critical finding was that even when the models did not predict the exact F-Measure value, they were highly effective at capturing the relative trends between different processing settings. This "trend-tracking" capability is essential for optimization tasks.

Figure 2: Comparison of true F-Measure responses against predictions from SVR, GP, ANN, and Ensemble models across test instances.

As illustrated in Figure 2, the surrogate models closely follow the fluctuations of the true F-Measure across various test instances. Furthermore, the SVR model demonstrated high computational efficiency, with an inference time of approximately 0.0039 seconds per document, making it suitable for real-time applications.

Discussion and Conclusion

The use of relative metrics as "proxies" allowed the models to generalize effectively even with a limited training set of 63 images. The researchers successfully demonstrated the utility of these models by using the predicted F-Measure as an objective function for Bayesian optimization, resulting in visually high-quality binarized images without human intervention.

Future work will focus on addressing outlier sensitivity in documents with extreme contrast variations. Additionally, the authors suggest exploring Convolutional Neural Networks (CNNs) to learn quality features directly from raw pixels, which may further improve accuracy if larger labeled datasets become available.

Summary updated

Abstract

Computation of document image quality metrics often depends upon the availability of a ground truth image corresponding to the document. This limits the applicability of quality metrics in applications such as hyperparameter optimization of image processing algorithms that operate on-the-fly on unseen documents. This work proposes the use of surrogate models to learn the behavior of a given document quality metric on existing datasets where ground truth images are available. The trained surrogate model can later be used to predict the metric value on previously unseen document images without requiring access to ground truth images. The surrogate model is empirically evaluated on the Document Image Binarization Competition (DIBCO) and the Handwritten Document Image Binarization Competition (H-DIBCO) datasets.
BibTeX
@inproceedings{singh2018learning,
  author = {Prashant Singh and Ekta Vats and Anders Hast},
  title = {{Learning Surrogate Models of Document Image Quality Metrics for Automated Document Image Processing}},
  booktitle = {DAS},
  year = {2018},
  url = {https://openreview.net/forum?id=wEyHQ86iBEW},
}
Figure