Restoring Degraded Old Films With Recursive Recurrent Transformer Networks

Shan Lin, Edgar Simo-SerraWACV 2024

TL;DR

This paper introduces Recursive Recurrent Transformer Network (RRTN), a novel framework for restoring degraded old films. RRTN leverages an explicit film noise mask estimator, second-order grid propagation with flow-guided deformable alignment, and an adaptive recursive architecture to effectively remove challenging film noise and improve overall image quality, outperforming existing methods.

What Problem Does This Paper Address?

Old films, despite their artistic and historical value, often suffer from significant degradation over time due to the analog medium. This deterioration, including dust, scratches, and fading, makes them difficult to view and preserve. While digital restoration techniques exist, manual restoration is labor-intensive and expensive, leading to a vast number of films remaining untouched. Existing automated restoration methods struggle with complex and varied film noises, leaving a gap for more effective and comprehensive solutions. This paper aims to bridge this gap by proposing a novel deep learning framework specifically designed for the challenging task of restoring degraded old films.

Methods

The proposed Recursive Recurrent Transformer Network (RRTN) tackles film restoration through several key innovations:

  • Explicit Film Noise Mask Estimation: Unlike previous methods that might implicitly handle noise or use single adjacent frames, RRTN explicitly estimates a film noise mask. It achieves this by warping both the previous and next frames to the current frame using optical flow, calculating their variations, and then computing the geometric mean of these absolute variations. This approach aims for a more accurate mask, crucial for guiding the restoration process.
  • Second-Order Grid Propagation and Flow-Guided Deformable Alignment: Inspired by existing work, RRTN employs a multi-directional grid propagation mechanism that allows for better utilization of temporal information from adjacent frames. Crucially, it incorporates "second-order propagation" to capture information over larger temporal ranges. This is combined with flow-guided deformable alignment, which aligns features from different frames more effectively, even in the presence of motion.
  • Adaptive Recursive Architecture: Recognizing that old films exhibit varying levels of degradation, RRTN features a recursive structure. This allows the network to iteratively refine the restoration process. If the output frames do not meet a certain quality threshold (determined by MSE), they are fed back into the network for further processing. This adaptive recursion enables the model to handle more difficult film noises without over-processing less degraded frames.

Datasets

The paper uses the following datasets:

  • REDS dataset: Used for generating synthetic degraded video data for training. Transformations are applied to simulate old film degradation.
  • DAVIS dataset: Used for generating test synthesized data.
  • Real old films: Collected from the internet for qualitative evaluation.

Key Insights and Ideas

  • Explicit Noise Masking is Key: The paper strongly advocates for an explicit film noise mask as a critical component for effective restoration, especially for complex noise patterns.
  • Leveraging Both Past and Future: Using both preceding and succeeding frames for noise mask estimation and temporal propagation significantly improves robustness compared to relying on only one temporal direction.
  • Recursion for Tough Cases: The adaptive recursive structure is vital for handling the diverse and severe degradations found in old films, allowing for deeper processing when needed.
  • Balancing Aggression and Gentleness: The use of two sets of trained parameters (one for aggressive initial processing and one for gentler refinement) is a clever way to manage the trade-off between aggressive noise removal and preserving image appearance.

Key Results

  • Quantitative Performance: RRTN achieves state-of-the-art results on both synthetic and real old film datasets, outperforming methods like DeepRemaster, BasicVSR++, and RVRT in terms of PSNR, SSIM, and LPIPS. On real films, it shows superior performance in no-reference metrics like BRISQUE and NIQE.
  • Qualitative Improvement: The paper presents compelling visual comparisons showing RRTN's ability to remove challenging large noise and significantly improve the overall quality of degraded films, particularly in areas where other methods struggle.
  • Inference Time: While RRTN's recursive nature leads to longer inference times compared to non-recursive methods, the authors argue this is acceptable given the manual effort saved and the superior quality achieved for difficult cases. The inference time scales linearly with the number of recursions.

Limitations and Future Work

  • Inference Time: The primary limitation acknowledged is the increased computation time due to the recursive architecture, especially for heavily degraded films requiring multiple recursions.
  • Noise Misclassification: The paper notes failure cases where non-noise elements (like smoke) can be mistakenly removed as noise, leading to undesired alterations in the output.
  • Future Directions: The authors suggest improving processing time by reducing model parameters and addressing the issue of noise misclassification to prevent the removal of important image content.

Summary updated

BibTeX
@inproceedings{lin2024restoring,
  author = {Shan Lin and Edgar Simo-Serra},
  title = {{Restoring Degraded Old Films With Recursive Recurrent Transformer Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure