Movie and video scale-time equalization application to flicker reduction

Julie Delon

TL;DR

This paper introduces "Scale-Time Equalization" (STE), a novel method for reducing flicker in movies and videos. By leveraging scale-space theory and a set of carefully defined axioms, STE offers a robust, global approach that preserves image dynamics and significantly improves visual quality, even in the presence of noise.

What Problem Does This Paper Address?

Image flicker, characterized by rapid variations in frame contrast and brightness, is a common artifact in old films and can also appear in modern videos due to factors like poor lighting control or low frame rates. While seemingly simple, effectively removing flicker without introducing new artifacts or distorting natural image variations is challenging. Existing methods often struggle with initialization sensitivity, lack of symmetry, or an inability to handle non-affine degradations. This paper aims to provide a more general, robust, and visually pleasing solution by treating flicker as a global, scale-dependent phenomenon.

Methods

The core of the paper is an axiomatic approach to defining a "de-flicker" operator. The authors propose a series of axioms:

  • Globality: Flicker is treated as a global image degradation, meaning transformations preserve the relative order of pixel intensities within each frame.
  • Figure/Background Independence: The method acts independently on different "level sets" of the image, allowing for distinct treatment of foreground and background elements if needed.
  • Time Shift Invariance & Symmetry: The de-flickering process should not depend on the starting point of the sequence or the direction of time.
  • Preservation of Constants: If a film consists of a constant image, the de-flickering process should leave it unchanged.

These axioms, particularly when applied to films of constant images, lead to the conclusion that the de-flickering process is equivalent to convolving the "rank function" (the inverse cumulative histogram) of each frame with a Gaussian kernel in time. This is the essence of Scale-Time Equalization (STE). The scale parameter of the Gaussian kernel controls the extent of temporal smoothing, allowing a trade-off between flicker reduction and the preservation of natural intensity variations.

Datasets

The paper evaluates the proposed method on sequences presenting:

  • Artificial Flicker: Synthetically generated sequences where flicker is added to a single base image with three distinct grey levels. This allows for controlled testing of the algorithm's behavior under specific degradation types.
  • Real Flicker: Extracts from historical films, specifically Chaplin's "His New Job" (1915) and "The Cure" (1917), which exhibit severe real-world flicker.

No new benchmarks were introduced; existing sequences were used for evaluation.

Key Insights and Ideas

  • Flicker as a Global Phenomenon: By treating flicker globally (based on histograms), the method becomes inherently robust to motion, shaking, and noise, which are often problematic for local approaches.
  • Scale-Space Theory for Temporal Smoothing: Applying scale-space theory (specifically, the heat equation via Gaussian convolution) to the temporal dimension of image histograms provides a principled way to smooth out rapid flicker variations while preserving slower, natural intensity changes.
  • Axiomatic Derivation: The rigorous derivation of the method from a set of fundamental axioms lends strong theoretical backing and ensures desirable properties like stability and robustness.
  • "Rank Function" as the Key: The insight that the de-flickering operation acts on the temporal evolution of each "rank function" (inverse cumulative histogram) is crucial. This allows for a unified treatment across different intensity levels.

Key Results

  • Visual Improvement: Experiments demonstrate a significant visual improvement in sequences after applying STE, with flicker becoming "almost imperceptible."
  • Robustness to Noise: The method shows remarkable robustness to both Gaussian and impulsive noise, with restored images remaining close to the original noise-free images. For instance, with Gaussian noise of standard deviation $\sigma=60$, the ratio of differences ||Unoise-Urestored||²/||Unoise - Unoise+flicker||² was still around 1/5.
  • Speed: The process is described as fast, with real-time performance achievable on standard hardware (e.g., 4 images/sec on an 800 MHz PC for 720x576 images).
  • Stability: The method exhibits stability, with the mean intensity of frames converging to a constant average value as the scale parameter increases, and the overall histograms of the sequence becoming very similar.

Limitations and Future Work

  • Scene Transitions: The authors acknowledge that STE loses its sense in the presence of scene transitions and suggest it should be associated with a "well chosen time-cut detector."
  • Quantization Noise: In cases of severe flicker and large scale equalization, an increase in quantization noise can be observed, especially in low-contrasted zones. The authors suggest bounding the slope of the contrast function as a potential mitigation.
  • Alternative Filters: While Gaussian convolution is derived from the axioms, the authors mention that other non-linear filters, like median filters, might be more adapted to scene changes.
  • Future Work: The paper implies that further exploration of alternative filters and integration with scene transition detection would be valuable.

Summary updated

Abstract

Image flicker is a general film effect, which can be observed in videos as well as in old films, and consists of fast variations of the frame contrast and brightness. Reducing flicker of a sequence improves its visual quality and can be an essential first treatment before ulterior manipulations. This paper presents an axiomatic analysis of the problem, which leads to a global and fast method of "de-flicker," based on the scale-space theory. The stability of this process, called scale-time equalization, is ensured by the scale-time framework. Results on different sequences are given and show great visual improvement.
BibTeX
@article{delon2006movie,
  author = {Julie Delon},
  title = {{Movie and video scale-time equalization application to flicker reduction}},
  journal = {IEEE Trans. Image Process.},
  year = {2006},
  url = {https://openreview.net/forum?id=2DOIqqX1i1},
}
Figure