Unsupervised Model-Based Learning for Simultaneous Video Deflickering and Deblotching

Anuj Fulari, Satish Mulleti, Ajit RajwadeWACV 2024

TL;DR

This paper introduces a novel, unsupervised approach to simultaneously remove flicker and blotch artifacts from videos. By combining a pixel-wise affine intensity model for flicker with a sparse representation for blotches, they achieve high-quality restoration through convex optimization, further accelerated by a learned neural network architecture. This method offers a robust and interpretable solution for enhancing vintage and high-speed video content.

What Problem Does This Paper Address?

Many videos, especially older ones recorded on magnetic tape or modern high-frame-rate footage under artificial lighting, suffer from visual artifacts that degrade their quality and hinder compression. Two prominent artifacts are:

  • Flicker: Rapid, often visually jarring, changes in average image intensity between consecutive frames. This can be caused by variations in exposure, chemical processing, film aging, or AC power fluctuations in lighting.
  • Blotches: Small, randomly located regions with undefined or corrupted pixel values, typically due to physical damage to the recording medium (e.g., dust, scratches on magnetic tape).

Existing methods often address these issues separately or rely on supervised learning, which can limit generalization to diverse video types. This paper aims to tackle both flicker and blotches simultaneously using an unsupervised, model-based approach that is both effective and interpretable.

Methods

The core of the proposed method lies in a model-based approach that formulates the problem as a joint optimization task.

  1. Flicker Modeling: Flicker is modeled using a pixel-wise affine intensity transformation between consecutive frames. The multiplicative and additive coefficients of this transformation are assumed to vary smoothly in space but randomly in time. To achieve spatial smoothness, these coefficients are represented using low-frequency Discrete Cosine Transform (DCT) bases.
  2. Blotch Modeling: Blotches are modeled as sparse artifacts, meaning they affect only a small number of pixels in any given frame.
  3. Joint Optimization: The paper proposes a convex optimization framework to simultaneously estimate the flicker coefficients (both multiplicative and additive) and the blotch regions. This is achieved by minimizing a cost function that includes terms for the reconstruction error, sparsity of flicker coefficients, and sparsity of blotches. The use of an $l_1$-norm regularization (LASSO) makes this a robust formulation.
  4. Handling Occlusions/Optical Flow Errors: To improve robustness, the method incorporates a mask that identifies and ignores pixels affected by occlusions or optical flow errors during the estimation process. This mask is constructed by analyzing the residual flow between forward and backward optical flow estimates.
  5. Learned FISTA (LFISTA): While the convex optimization approach is powerful, it can be computationally expensive for video processing. To address this, the authors adapt the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA) by unrolling its iterations into a deep neural network architecture. This "Learned FISTA" (LFISTA) allows for significant speedups while retaining interpretability. Key components of FISTA, like the DCT basis matrix, can also be learned within this framework.

Datasets

The authors evaluated their method on three datasets:

  • Dataset D1 (10 videos): A collection of vintage movie clips, including sequences from "Le mort qui tue" (1913/14), a sequence from [2], and videos shared by authors of [31].
  • Dataset D2 (4 videos): High frame rate videos (150-190 fps) exhibiting flicker, sourced from [25].
  • Dataset D3 (41 videos): A diverse collection of YouTube videos containing both blotch and flicker artifacts.

The LFISTA network was trained on synthetically degraded video clips, distinct from the evaluation datasets.

Key Insights and Ideas

  • Simultaneous Restoration: The core insight is that flicker and blotches can be effectively addressed jointly within a single, unified model, rather than as separate post-processing steps. This allows the model to leverage the interplay between these artifacts.
  • Sparsity and Smoothness Priors: The effectiveness of the model hinges on exploiting prior knowledge: flicker coefficients are spatially smooth, while blotches are spatially sparse. These priors are mathematically encoded into the optimization problem.
  • Model-Based Learning: The paper bridges the gap between traditional model-based computer vision and modern deep learning. By unrolling a known optimization algorithm (FISTA) into a neural network, they achieve the benefits of both: interpretability and robustness from the model, and efficiency from learned deep networks.
  • Unsupervised Approach: The method's unsupervised nature is a significant advantage, as it doesn't require ground truth restored videos for training or testing, making it applicable to real-world scenarios where such data is scarce.

Key Results

  • Superior Performance: The proposed LFISTA method consistently outperforms state-of-the-art techniques like RTN [31] and DeepRemaster [13] on temporal quality metrics (SM, OFE1, OFE2, OFEP). It also shows strong performance against NFFA [18].
  • Effective Blotch Removal: Qualitative results (Figure 4) demonstrate superior blotch removal compared to other methods.
  • Computational Efficiency: LFISTA significantly reduces processing time. For a typical video, FISTA took around 10 hours, while LFISTA reduced this to approximately 5 minutes.
  • Robustness: The method is robust to optical flow errors and occlusions, thanks to the masking strategy.
  • No Explicit Training for NFFA: Notably, LFISTA outperforms NFFA [18] despite NFFA requiring no explicit training, highlighting the strength of the model-based approach.

Limitations and Future Work

  • Computational Cost: While significantly faster than FISTA, LFISTA is still not real-time. The authors suggest that further improvements in unrolling techniques could enhance computational efficiency.
  • Motion Boundaries: The method's performance on blotches located at motion boundaries could be improved.
  • Other Artifacts: Vintage videos can also suffer from other artifacts like non-rigid jitters. The authors suggest a future direction could be a combined engine to address flicker, blotches, and jitters simultaneously.
  • Color Videos: For color videos, restoration was performed on the V channel of the HSV color space, leaving H and S intact. Future work could explore more sophisticated color restoration.

Summary updated

BibTeX
@inproceedings{fulari2024unsupervised,
  author = {Anuj Fulari and Satish Mulleti and Ajit Rajwade},
  title = {{Unsupervised Model-Based Learning for Simultaneous Video Deflickering and Deblotching}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure