Optimization-Based Data Generation for Photo Enhancement

Mayu Omiya, Yusuke Horiuchi, Edgar Simo-Serra, Satoshi Iizuka, Hiroshi IshikawaWorkshops @ CVPR 2019 · DOI

Introduction

Training deep learning models for photo enhancement typically requires massive datasets of paired "before and after" images. However, producing these pairs manually is both expensive and time-consuming. While datasets like MIT-Adobe FiveK exist, they often lack the scale necessary for modern deep learning architectures. This paper presents an improved automated pipeline based on Black Box Model Optimization (BBMO) to generate high-quality synthetic training data from readily available online photos.

The Data Generation Pipeline

The core challenge in synthetic data generation for photo enhancement is "reverse-engineering" the parameters of a photo editor. The proposed framework takes high-quality images and applies random "de-enhancement" settings to create low-quality versions. The goal is then to find the exact parameters that can restore the original high-quality look.

The data generation pipeline: high-quality images are de-enhanced, filtered for naturalness, and then reconstructed using optimization to find the best enhancement parameters.

The authors introduce several key improvements to the standard BBMO workflow:

  1. Naturalness Filtering: A VGG19-based classifier is used to discard de-enhanced images that look "unnatural," ensuring the model learns from realistic degradation.
  2. Robust Optimization: The system uses Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to find restoration parameters, replacing the less reliable Nelder-Mead algorithm.
  3. Perceptual Similarity Metric: Instead of relying solely on Mean Squared Error (MSE), the pipeline uses a learned similarity metric incorporating RGB, Lab, and HSV color spaces along with the Structural Similarity Index (SSIM) to prune failed optimizations.

Optimization and Results

The choice of optimization algorithm is critical because the quality of the synthetic data depends on how well the optimizer can "undo" the de-enhancement. CMA-ES proves significantly more robust than previous methods at finding global optima in the complex, non-convex parameter space of a 21-slider photo editor.

Visual comparison of reconstruction quality: (a) De-enhanced input, (b) Nelder-Mead reconstruction, (c) CMA-ES reconstruction, and (d) Original high-quality image.

Quantitatively, CMA-ES achieved an MSE of 0.0025 and an SSIM of 0.9328, significantly outperforming the Nelder-Mead approach (MSE 0.0105, SSIM 0.8799). This higher-quality data allowed a model trained on only 3,224 images to outperform a previous BBMO model trained on over 14,000 images. In perceptual studies, users preferred images enhanced by this model over the original inputs 89% of the time.

Conclusion

By combining robust derivative-free optimization with a multi-space color similarity metric, this research demonstrates that high-quality synthetic data can be generated effectively for photo enhancement tasks. This approach reduces the reliance on expensive manual labeling while maintaining high perceptual quality in the final enhanced results.

Summary updated

BibTeX
@inproceedings{omiya2019optimization,
  author = {Mayu Omiya and Yusuke Horiuchi and Edgar Simo-Serra and Satoshi Iizuka and Hiroshi Ishikawa},
  title = {{Optimization-Based Data Generation for Photo Enhancement}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00240},
}
Figure