Extracting Vignetting and Grain Filter Effects From Photos

Abdelrahman Abdelhamed, Jonghwa Yim, Abhijith Punnappurath, Michael S. Brown, Jihwan Choe, Kihwan KimWACV 2022

TL;DR

This paper introduces an efficient method to extract and replicate two common, spatially varying photo filter effects: vignetting and image grain. By using lightweight neural networks to estimate filter parameters and efficient regeneration techniques, their approach achieves a good balance of realism and speed, making it suitable for real-time deployment on smartphones.

What Problem Does This Paper Address?

Modern smartphones offer a wide array of real-time camera filters that enhance photos with desired visual effects. While many filters focus on color and tonal adjustments, a significant category of filters imparts spatially varying effects, such as vignetting (darkening of image corners) and image grain (simulating the texture of photographic film). Existing methods for filter extraction often focus solely on color or style transfer, failing to capture these specific spatial characteristics. This leaves a gap in being able to precisely extract and replicate these common, yet distinct, visual enhancements from a single example image.

Methods

The core of this paper lies in a two-pronged approach for extracting and applying vignetting and grain filters:

  • Filter Parameter Estimation: For both vignetting and grain, the authors treat parameter estimation as a classification problem. They train lightweight Convolutional Neural Networks (CNNs) to predict discrete levels of vignetting strength and grain intensity from an input image that has already been stylized.
  • Vignetting: They model vignetting using an isotropic 2D Gaussian filter. To improve prediction accuracy, they augment the input image with "radial intensity bands" – features derived from average pixel intensities in ring-shaped areas around the image center.
  • Grain: They simulate grain using OpenSimplex noise. The CNN for grain intensity is trained on a range of intensities.
  • Filter Regeneration and Application: Once the filter parameters are estimated, the method regenerates the corresponding filter.
  • Vignetting: A 2D Gaussian filter is generated based on the predicted strength.
  • Grain: Simplex noise is generated with the predicted intensity. These regenerated filters are then applied to a new content image. The application of vignetting is a multiplicative operation, while grain is an additive one.

Datasets

  • Training/Validation/Testing: The DIV2K dataset was used for training, validation, and testing.
  • Synthetic Data Generation: The authors generated synthetic vignetted and grainy images to train their respective classifiers, addressing the lack of readily available paired datasets.

Key Insights and Ideas

  • Spatially Varying Filters are Distinct: The paper highlights that vignetting and grain are specific spatial effects that require dedicated extraction methods, distinct from general color or style transfer.
  • Classification for Efficiency: Framing filter parameter estimation as a classification problem, rather than regression, allows for the use of simpler, more efficient CNN architectures.
  • Radial Intensity Bands Boost Accuracy: Augmenting the input with radial intensity bands significantly improves the accuracy of vignetting strength prediction, demonstrating the importance of feature engineering for specific spatial effects.
  • Trade-off Between Realism and Efficiency: The method achieves a good balance by using efficient regeneration techniques (Gaussian filters, Simplex noise) and lightweight CNNs, making it practical for real-time mobile applications.

Key Results

  • Vignetting Accuracy: The vignetting strength classifier achieves 92% accuracy, with precision, recall, and F1 scores around 0.92.
  • Grain Accuracy: The grain intensity classifier achieves 95% accuracy, with precision, recall, and F1 scores around 0.95.
  • Efficiency: The method is remarkably fast, with vignetting extraction taking approximately 7 ms and grain extraction taking 77 ms per megapixel on a Tesla V100 GPU. This is significantly faster than existing style transfer methods.
  • Qualitative Results: The paper demonstrates successful extraction and transfer of both vignetting and grain effects, visually replicating them on new images with high fidelity. Comparisons show their method outperforms general color and style transfer methods in preserving these specific spatial effects.
  • Deployment: The method's efficiency has led to its deployment on millions of flagship smartphones.

Limitations and Future Work

  • Vignetting Strength Granularity: The paper notes that distinguishing between very similar vignetting strengths (e.g., 5 and 6) can be challenging, leading to slightly lower accuracy in those ranges. They omitted training for strengths below 5 due to this difficulty.
  • Future Work: The authors suggest that their method can be combined with other color transfer methods to achieve a more comprehensive filter stylization pipeline, as demonstrated in their figures. The efficient nature of their approach opens doors for further integration and exploration of more complex spatial effects.

Summary updated

BibTeX
@inproceedings{abdelhamed2022extracting,
  author = {Abdelrahman Abdelhamed and Jonghwa Yim and Abhijith Punnappurath and Michael S. Brown and Jihwan Choe and Kihwan Kim},
  title = {{Extracting Vignetting and Grain Filter Effects From Photos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}

Loading related content...

Figure