ROMNet: Renovate the Old Memories

Runsheng Xu, Zhengzhong Tu, Yuanqi Du, Xiaoyu Dong, Jinlong Li, Zibo Meng, Jiaqi Ma, Hongkai YuarXiv 2022 · OpenReview

TL;DR

This paper introduces ROMNet, a novel end-to-end deep learning framework that simultaneously restores and colorizes degraded old photos. By leveraging a reference image and a multi-scale spatial-preserving color histogram fusion method, ROMNet significantly reduces the need for large training datasets, a major bottleneck in previous approaches.

What Problem Does This Paper Address?

Renovating old photographs is a challenging task due to the commingled degradations they suffer, such as cracks, noise, and color fading. Traditional methods often require manual intervention by experts, making them labor-intensive and time-consuming. While deep learning has shown promise in image restoration and colorization, existing methods typically demand massive, paired datasets for training. This is particularly problematic for old photo restoration, where creating such large-scale, high-quality datasets is extremely difficult and expensive. Furthermore, most prior work focuses on either restoration or colorization, but not both simultaneously, leading to suboptimal results when applied to real-world degraded photos.

Methods

ROMNet is an end-to-end framework composed of three key modules:

  • Restoration Sub-network: This module tackles various degradations (cracks, noise, blur, etc.) in the luminance channel of the input image. It employs a multi-level Residual Dense Network (RDN) architecture. The multi-level design allows the network to capture both local details and larger structural information, enhancing its ability to repair complex flaws.
  • Similarity Sub-network: This module establishes a correspondence between the input image and a reference color image. It utilizes a pre-trained ResNet34 to extract multi-scale feature maps from both images. Crucially, it introduces a learnable coefficient to weight these feature maps, creating multi-scale similarity maps that are then used to align the reference image's color information with the input. This differs from prior work that often uses single-scale similarity maps.
  • Colorization Sub-network: This module predicts the chrominance channels (a and b in the CIE Lab color space) of the image. It incorporates a novel Spatial-Preserve color Histogram (SPHist) method, which captures both color statistics and spatial information from the reference image. SPHist is differentiable, allowing for end-to-end training. The warped SPHist, guided by the multi-scale similarity maps, is fused with intermediate features in a U-Net-like backbone to predict the final color output. This multi-scale fusion approach is a key innovation.

The training objective is a weighted sum of several losses, including luminance reconstruction loss, perceptual loss, histogram loss (EMD), chroma reconstruction loss, and an adversarial loss to enhance visual quality.

Datasets

The paper introduces the RealOld dataset, which is claimed to be the first public, real-world old photo dataset with paired ground truth for evaluation. It consists of 200 authentic grayscale old photos, each paired with a manually restored and colorized "pristine" image created by Photoshop experts.

In addition to RealOld, the authors also use:

  • DIV2K: A standard dataset for image super-resolution, used here to evaluate the colorization sub-network's performance on images without degradation.
  • Pascal VOC: Used to generate synthetic degraded images by adding noise, blur, and simulating damage. This dataset is used for training and evaluating the full restoration and colorization pipeline.

Key Insights and Ideas

  • Reference-based colorization significantly reduces data dependency: By leveraging a reference image, the model can achieve good colorization results with far less training data than methods that rely solely on large, paired datasets. This is a crucial step towards making advanced image restoration and colorization more accessible.
  • Multi-scale feature fusion is key for robust similarity matching: Using similarity maps at multiple scales, weighted by learnable coefficients, allows the model to better capture correspondences between the input and reference images, leading to more accurate color transfer.
  • Spatial-Preserve color Histogram (SPHist) bridges color statistics and spatial information: This novel histogram representation is differentiable and effectively encodes both color distribution and spatial layout, providing a strong prior for colorization.
  • Joint restoration and colorization is essential for real-world old photos: Separately addressing degradation and colorization is insufficient. ROMNet's integrated approach effectively handles both, producing more visually pleasing and accurate results.
  • The RealOld dataset is a valuable contribution: Providing a real-world dataset with expert-annotated ground truth is critical for advancing research in this domain, enabling more realistic evaluations.

Key Results

  • Quantitative Performance: ROMNet achieves state-of-the-art results on both synthetic (DIV2K, Pascal VOC) and the newly introduced RealOld datasets, outperforming existing methods in PSNR, SSIM, and LPIPS metrics. For instance, on the RealOld dataset, ROMNet achieved the best scores (17.20 PSNR, 0.758 SSIM, 0.258 LPIPS), demonstrating its resilience to real-world degradations.
  • Qualitative Performance: Visual comparisons show that ROMNet effectively fixes degradations and produces vivid, natural-looking colors, often surpassing other methods in terms of visual quality and detail preservation.
  • User Study: A user study on the RealOld dataset indicated that ROMNet was selected as the top-ranked result by 50.6% of users, significantly outperforming other methods.
  • Robustness to Reference Selection: The model shows robustness to the choice of reference image, with only slight performance drops even when using less similar references.

Limitations and Future Work

  • Reference Selection: While the model is robust, the paper acknowledges that the quality of the reference image still impacts the final output. Future work could explore more sophisticated automatic reference selection mechanisms.
  • Computational Cost: Although ROMNet is designed to be data-efficient, the computational cost of training and inference, especially with the multi-level RDN and multi-scale similarity computations, might still be a consideration for real-time applications.
  • Generalization to Extreme Degradations: While effective on common old photo degradations, the model's performance on extremely severe or unusual types of damage might require further investigation.

The authors suggest that their work opens doors for further research in deep learning-based old photo restoration and enhancement, particularly with the availability of the RealOld dataset.

Summary updated

Abstract

Renovating the memories in old photos is an intriguing research topic in computer vision fields. These legacy images often suffer from severe and commingled degradations such as cracks, noise, and color-fading, while lack of large-scale paired old photo datasets makes this restoration task very challenging. In this work, we present a novel reference-based end-to-end learning framework that can jointly repair and colorize the degraded legacy pictures. Specifically, the proposed framework consists of three modules: a restoration sub-network for degradation restoration, a similarity sub-network for color histogram matching and transfer, and a colorization subnet that learns to predict the chroma elements of the images conditioned on chromatic reference signals. The whole system takes advantage of the color histogram priors in a given reference image, which vastly reduces the dependency on large-scale training data. Apart from the proposed method, we also create, to our knowledge, the first public and real-world old photo dataset with paired ground truth for evaluating old photo restoration models, wherein each old photo is paired with a manually restored pristine image by PhotoShop experts. Our extensive experiments conducted on both synthetic and real-world datasets demonstrate that our method significantly outperforms state-of-the-arts both quantitatively and qualitatively.
BibTeX
@misc{xu2022romnet,
  author = {Runsheng Xu and Zhengzhong Tu and Yuanqi Du and Xiaoyu Dong and Jinlong Li and Zibo Meng and Jiaqi Ma and Hongkai Yu},
  title = {{ROMNet: Renovate the Old Memories}},
  year = {2022},
  eprint = {2202.02606},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2202.02606},
}
Figure