TL;DR
This paper introduces Deep Photo Scan (DPScan), a novel semi-supervised learning framework for restoring degraded smartphone-scanned photos. By creating a new real-world degradation dataset (DIV2K-SCAN), proposing a "Local Alignment" technique, and leveraging generative adversarial networks for domain generalization, DPScan achieves state-of-the-art results in both single and multiple degradation scenarios.
What Problem Does This Paper Address?
Digitizing physical photographs using smartphone scanning apps has become ubiquitous, offering convenience and easy sharing. However, the resulting digital images often suffer from significant degradation compared to professionally scanned or original digital photos. This degradation stems from various factors including low-cost smartphone cameras, lens imperfections, printing artifacts, and the inherent challenges of capturing 3D objects in a 2D image, leading to structural misalignments. Existing methods often rely on simulating degradations, which don't fully capture the complexity of real-world smartphone scans. Furthermore, acquiring large, perfectly aligned datasets for supervised training is labor-intensive and costly, limiting the effectiveness of purely supervised approaches. This paper aims to bridge this gap by developing a robust restoration method that can handle real-world degradations and generalize across various scanning conditions, even with limited labeled data.
Methods
The DPScan framework is built upon several key innovations:
- DIV2K-SCAN Dataset: The authors introduce a new dataset specifically designed to capture real-world smartphone scanning degradations. This dataset is created by printing high-quality images, scanning them with smartphones under controlled conditions, and then carefully processing them.
- Local Alignment (LA): To address the structural misalignments that persist even after global alignment, LA is proposed. This technique refines the alignment of scanned patches to their ground truth, improving restoration quality and the reliability of quantitative evaluation.
- Domain Generalization through Simulation: To enhance the model's ability to handle diverse scanning conditions (different phones, lighting, etc.), the authors simulate various degradation styles using low-level image transformations. This allows the model to generalize beyond the specific degradations present in the training data.
- Semi-Supervised Learning (SSL): To overcome the limitations of scarce labeled data, DPScan employs a semi-supervised approach. It leverages Generative Adversarial Networks (GANs) to generate "pseudo-scanned" images from unscanned high-quality images. This allows the restoration network to be trained on a much larger, more diverse dataset, balancing supervised and unsupervised losses.
- Network Architecture (RECA): The core restoration network (G1) is based on a U-Net architecture, enhanced with Residual Efficient Channel Attention (RECA) blocks. This customization aims to improve learning capabilities and efficiency.
Datasets
- DIV2K-SCAN: A newly introduced dataset for smartphone-scanned photo restoration, containing real-world degradations. It includes 800 scanned image pairs for supervised training, validation, and testing.
- Flickr2K: A dataset of 2,600 unscanned high-quality images used for the semi-supervised learning component to diversify training data.
Key Insights and Ideas
- Real-world degradation is complex: The paper highlights that smartphone scanning involves more intricate degradations than simple simulated noise or blur, necessitating specialized datasets and methods.
- Local alignment is crucial: Even with global alignment, subtle structural misalignments significantly impact restoration quality and evaluation metrics. LA proves effective in mitigating this.
- Semi-supervised learning is powerful for data scarcity: By leveraging GANs to create synthetic data from unlabeled images, DPScan effectively expands its training set, leading to better generalization.
- Domain generalization is key for practical use: A model trained on one set of smartphone scans may not perform well on others. Simulating diverse degradation styles is essential for robust performance across different devices and environments.
Key Results
- Quantitative Improvement: DPScan achieves state-of-the-art performance, outperforming previous works and industrial products. For instance, on the 1-domain DIV2K-SCAN (iPhone XR), DPScan achieves a PSNR of 25.26, LPIPS of 0.1242, and MS-SSIM of 0.9446.
- Generalization Performance: The generalized DPScan (G-DPScan) shows strong performance across unseen domains (color-balanced iPhone XR, Xperia XZ1), achieving an average PSNR of 23.17, LPIPS of 0.1780, and MS-SSIM of 0.9223.
- Qualitative Results: DPScan produces visually superior results, characterized by clearer edges, reduced artifacts, and better preservation of fine details compared to existing methods and industrial products. It effectively removes haze and color fading.
Limitations and Future Work
- Training Time: The paper notes that achieving the best generalization performance (with K=100 simulated domains) requires a long training time.
- Computational Cost: While RECA aims for efficiency, deep learning models can still be computationally intensive.
- Future Work: The authors suggest further exploration of deep modules for DPScan, pseudo-scanned photo synthesis, and the impact of the number of simulated domains (K) on generalization. The current work provides a strong baseline for future research in smartphone photo scanning restoration.
Summary updated