Abstract
This paper introduces a novel method to predict image inpainting quality by calculating the expected reconstruction error of a Wiener filter. Unlike previous methods that focus on human visual saliency, this approach provides a mathematically grounded "difficulty metric" that works both before and after data loss. By identifying regions that are statistically unique and thus hard to replace, this method serves as a powerful tool for intelligent image compression and error concealment.
Introduction
Inpainting—the process of "filling in" missing or corrupted parts of an image—is a staple of modern image processing. However, a significant challenge remains: predicting how successful an inpainting attempt will be before it is performed. Existing approaches generally fall into two categories: saliency estimation, which identifies what humans find visually important, and post-hoc evaluation, which measures quality only after the inpainting is complete. Neither approach is suitable for deciding which parts of an image to discard during compression to ensure they can be recovered later.
To address this, we propose a universal quality measure that mathematically identifies which image regions are statistically unique versus those that are redundant and easily reconstructed from their surroundings.
Methodology
The core of the proposed method is built on the Wiener Filter, the optimal linear estimator for least-squares reconstruction. The authors derive a formula for the expected Mean Square Error (MSE) of a Wiener filter, showing that this error can be estimated even when the original pixels are unknown by analyzing the correlation between surrounding "known" patches.
Two-Way Prediction
- Before Loss: By treating every block as a potential "missing" area, the method creates an Inpainting Quality Map. This map highlights regions that should not be removed because they are too unique to be recovered.
- After Loss: If an image is already corrupted (e.g., due to packet loss), the same mathematical framework predicts which holes will be the hardest to fix.
The implementation uses a sliding window to calculate the local correlation matrix and the resulting expected error, producing a heatmap where brighter areas indicate higher reconstruction difficulty.
Experimental Results
The researchers evaluated their method using the Bruce et al. Benchmark (120 test images) and compared it against six well-known saliency methods, including Itti98, GBVS07, and AIM09.
Performance Metrics
- Superior Reconstruction: When using the proposed map to guide block removal for compression, the resulting MSE was significantly lower than when using saliency-based maps. For instance, the proposed method achieved an MSE of $1.01 \times 10$, while the popular Itti98 method resulted in $4.07 \times 10$.
- High Correlation: The predicted quality map showed a very high correlation (Pearson and Spearman coefficients often $> 0.9$) with the actual MSE of the final reconstructed images, proving the accuracy of the prediction.
- Subjective Quality: In a study with 17 human subjects, the proposed method's reconstructions received a mean score of 4.04/5, significantly outperforming all six comparative saliency-based methods.
Discussion
A key insight of this research is that reconstruction difficulty is a distinct property from visual importance. A highly detailed texture might be visually salient, but if it is repetitive, a Wiener filter can reconstruct it easily. Conversely, a single unique edge might be "low saliency" but impossible to inpaint if removed.
The method relies on the stationarity assumption, positing that an image has similar statistical features in different parts. If a missing area can be "explained" by the statistics of the known areas, the predicted error remains low.
Limitations and Future Work
The model currently exhibits "structural blindness," as it is purely statistical and does not account for complex semantic objects. If a unique edge is removed without similar local data, the model may underestimate difficulty. Additionally, the assumption that the correlation matrix of the missing part matches the known parts does not always hold in diverse textures. Future iterations could replace MSE with more perceptually-optimized quality measures to better reflect human visual judgment.
Summary updated