Automatic Photo Adjustment Using Deep Neural Networks

Zhicheng Yan, Hao Zhang, Baoyuan Wang, Sylvain Paris, Yizhou YuACM Trans. Graphics 2016 · OpenReview

TL;DR

This paper introduces a deep learning framework for automatic photo adjustment that can learn complex, semantically-aware artistic styles. By incorporating novel contextual features derived from scene parsing and object detection, their method achieves state-of-the-art results, outperforming previous approaches in capturing nuanced, spatially varying enhancements.

What Problem Does This Paper Address?

Manually retouching photos to achieve artistic and dramatic visual impressions is a time-consuming and skill-intensive process. While automated photo adjustment tools exist, they often struggle with the subtle, content-dependent, and spatially varying nature of artistic styles. Many styles rely on an artist's understanding of image semantics (e.g., enhancing a person's face differently from the sky), which traditional algorithms fail to capture. This paper aims to bridge this gap by developing an automated system that can learn and apply these sophisticated artistic styles from examples.

Methods

The core of the proposed method is a Deep Neural Network (DNN) trained to perform photo adjustment as a regression problem. Instead of directly predicting the final pixel colors, the DNN predicts parameters for a color transform. This approach separates the high-frequency color variations from the underlying spatial adjustments, allowing the DNN to focus on learning smoother, more complex mappings.

Key methodological contributions include:

  • Deep Neural Network (DNN) for Regression: A multi-layer DNN is employed as a universal approximator to learn the highly nonlinear and spatially varying color mapping function.
  • Color Transforms: The output of the DNN is a set of coefficients for a color transform (e.g., quadratic color transforms). This allows the model to absorb high-frequency color variations, making the DNN's task of learning the spatial adjustments more manageable.
  • Novel Contextual Feature Descriptor: This is a significant contribution. The descriptor captures the distribution of semantic categories (like "sky," "car," "person") within local image regions at multiple scales. It's constructed by first performing scene parsing and object detection to create a semantic label map, and then using a multiscale spatial pooling schema to build histograms of these semantic labels around each pixel. This allows the model to understand and adjust based on the "meaning" of different image regions.
  • Pixelwise and Global Features: In addition to contextual features, the system also uses standard pixelwise features (color, position) and global image features (intensity distribution, scene brightness) as input to the DNN.
  • Superpixel-Based Sampling: To manage the vast number of pixels and avoid biased training data, images are first segmented into superpixels, and training samples are collected from these regions.
  • Cross-Entropy Based Training Set Selection: An algorithm is proposed to select a representative subset of training images from a large collection, optimizing for feature space coverage and reducing the need for extensive manual annotation.

Datasets

  • Custom Datasets: The authors created three distinct local enhancement style datasets by having a professional photographer retouch 115 images. These datasets represent "Foreground Pop-Out," "Local Xpro," and "Watercolor" effects.
  • MIT-Adobe FiveK Dataset: This dataset, containing 5000 raw images retouched by five photographers, was used for learning global adjustments and for comparison with other methods.
  • Instagram Dataset: A subset of 50 images from the MIT-Adobe FiveK dataset was used to learn Instagram's popular filter effects.

Key Insights and Ideas

  • Semantics are Crucial for Artistic Enhancement: The paper strongly demonstrates that understanding image semantics (what objects are present and where) is essential for replicating sophisticated artistic photo adjustments. Simply analyzing pixel colors or global statistics is insufficient.
  • Deep Learning Excels at Complex Mappings: DNNs prove to be highly effective at learning the intricate, nonlinear relationships required for artistic photo enhancement, especially when combined with rich feature representations.
  • Contextual Features are Powerful: The novel multiscale contextual features, derived from semantic analysis, significantly improve the quality of local adjustments, leading to results much closer to ground truth compared to methods without them.
  • Separating Color Transforms from Spatial Adjustments is Beneficial: By predicting color transform parameters rather than direct pixel values, the DNN can focus on learning the more challenging spatial aspects of the adjustment.
  • Efficient Training Data Selection Matters: The cross-entropy based method for selecting training data is shown to be effective, especially when dealing with limited datasets, leading to better generalization.

Key Results

  • Quantitative Improvement: The proposed method achieved significantly lower testing errors (mean L2 error) compared to previous methods like [Wang et al. 2011] and [Hwang et al. 2012] across various datasets and stylistic effects. For instance, contextual features reduced the mean L2 error by approximately 20% for the tested local effects.
  • Qualitative Superiority: Visual comparisons consistently show that the authors' enhanced images are closer to the ground truth (photographer-created) than those produced by competing methods. This is particularly evident in capturing spatially varying adjustments and avoiding artifacts.
  • User Study Validation: Two user studies confirmed the superiority of their approach. Participants preferred the authors' results over those from [Hwang et al. 2012] in blind comparisons, and statistical tests indicated a significant improvement in visual quality and adherence to ground truth.
  • Generalization: The trained models demonstrated good generalization capabilities, producing plausible adjustments even on novel testing images with significant visual differences from the training data.

Limitations and Future Work

  • Reliance on Semantic Analysis: The performance of the contextual features is directly dependent on the accuracy of the underlying scene parsing and object detection algorithms. Errors in these upstream tasks can propagate and lead to incorrect adjustments.
  • "Black Box" Nature of DNNs: The authors acknowledge that DNNs are often treated as black boxes, and interpreting how they combine features and make predictions is an ongoing research area.
  • Time-Consuming DNN Architecture Search: Finding the optimal DNN architecture (number of layers, neurons) can be a time-consuming trial-and-error process.
  • Future Work: The authors suggest exploring alternative semantic descriptor designs and further investigating the interpretability of deep neural networks in this context. They also note that their framework can be readily applied to global image adjustments, which they demonstrate.

Summary updated

Abstract

Photo retouching enables photographers to invoke dramatic visual impressions by artistically enhancing their photos through stylistic color and tone adjustments. However, it is also a time-consuming and challenging task that requires advanced skills beyond the abilities of casual photographers. Using an automated algorithm is an appealing alternative to manual work, but such an algorithm faces many hurdles. Many photographic styles rely on subtle adjustments that depend on the image content and even its semantics. Further, these adjustments are often spatially varying. Existing automatic algorithms are still limited and cover only a subset of these challenges. Recently, deep learning has shown unique abilities to address hard problems. This motivated us to explore the use of deep neural networks (DNNs) in the context of photo editing. In this article, we formulate automatic photo adjustment in a manner suitable for this approach. We also introduce an image descriptor accounting for the local semantics of an image. Our experiments demonstrate that training DNNs using these descriptors successfully capture sophisticated photographic styles. In particular and unlike previous techniques, it can model local adjustments that depend on image semantics. We show that this yields results that are qualitatively and quantitatively better than previous work.
BibTeX
@article{yan2016automatic,
  author = {Zhicheng Yan and Hao Zhang and Baoyuan Wang and Sylvain Paris and Yizhou Yu},
  title = {{Automatic Photo Adjustment Using Deep Neural Networks}},
  journal = {ACM Trans. Graphics},
  year = {2016},
  url = {https://openreview.net/forum?id=l20p0xV9io4r},
}
Figure