Abstract
Computation of document image quality metrics often depends upon the availability of a ground truth image corresponding to the document. This limits the applicability of quality metrics in applications such as hyperparameter optimization of image processing algorithms that operate on-the-fly on unseen documents. This work proposes the use of surrogate models to learn the behavior of a given document quality metric on existing datasets where ground truth images are available. The trained surrogate model can later be used to predict the metric value on previously unseen document images without requiring access to ground truth images. The surrogate model is empirically evaluated on the Document Image Binarization Competition (DIBCO) and the Handwritten Document Image Binarization Competition (H-DIBCO) datasets.
Introduction
In the field of document digitization, binarization—the conversion of gray-scale or color scans into clean black-and-white images—is a fundamental preprocessing step. The performance of binarization algorithms is highly dependent on hyperparameter tuning. Traditionally, this tuning is performed by comparing the algorithm's output against a manually curated "ground truth" image using metrics such as F-Measure or Peak Signal-to-Noise Ratio (PSNR).
However, a significant challenge arises in real-world production environments where ground truth images for new documents are unavailable. This creates a circular dependency: one cannot calculate the quality metric without a reference, yet the metric is required to automatically optimize the processing algorithm. This research addresses this problem by developing surrogate models that can estimate these quality metrics without requiring reference images.
Methodology
The proposed approach treats the quality metric as a function to be approximated through surrogate modeling (or metamodeling). This involves training a computationally efficient mathematical model to mimic the behavior of a complex metric like the F-Measure.
Training Pipeline
The training process utilizes existing datasets where ground truth is available. As shown in Figure 1, the system computes relative image quality metrics (I) by comparing the original degraded images (X) with their processed versions (P). These relative metrics—such as PSNR, Distance Reciprocal Distortion (DRD), and Negative Rate Metric (NRM)—serve as features for the surrogate model. The target for training is the actual F-Measure calculated using the ground truth images (G).
Model Architectures
Three distinct regression architectures were evaluated:
- Support Vector Regression (SVR): Utilizing a Sequential Minimal Optimization (SMO) solver.
- Artificial Neural Networks (ANN): Specifically, feed-forward back-propagation networks.
- Gaussian Processes (GP): Stochastic models that provide both predictions and uncertainty measures.
Experimental Setup and Results
The study utilized 86 images from the DIBCO (2009, 2011, 2013) and H-DIBCO (2010, 2012, 2014, 2016) datasets. These documents contain various historical degradations, including ink bleed-through, stains, and low contrast. 63 images were used for training, while 23 were reserved for testing.
Performance Analysis
Support Vector Regression (SVR) emerged as the most robust model, achieving the lowest Root Relative Square Error (RRSE) of 0.8053. A critical finding was that even when the models did not predict the exact F-Measure value, they were highly effective at capturing the relative trends between different processing settings. This "trend-tracking" capability is essential for optimization tasks.
As illustrated in Figure 2, the surrogate models closely follow the fluctuations of the true F-Measure across various test instances. Furthermore, the SVR model demonstrated high computational efficiency, with an inference time of approximately 0.0039 seconds per document, making it suitable for real-time applications.
Discussion and Conclusion
The use of relative metrics as "proxies" allowed the models to generalize effectively even with a limited training set of 63 images. The researchers successfully demonstrated the utility of these models by using the predicted F-Measure as an objective function for Bayesian optimization, resulting in visually high-quality binarized images without human intervention.
Future work will focus on addressing outlier sensitivity in documents with extreme contrast variations. Additionally, the authors suggest exploring Convolutional Neural Networks (CNNs) to learn quality features directly from raw pixels, which may further improve accuracy if larger labeled datasets become available.
Summary updated