Crowd-Powered Photo Enhancement Featuring an Active Learning Based Local Filter

Satoshi Kosugi, Toshihiko Yamasaki

TL;DR

This paper introduces a novel crowd-powered photo enhancement method that achieves content-aware local adjustments by leveraging active learning to efficiently guide human annotators. By intelligently selecting key pixels and predicting parameters for others, their approach significantly improves image quality and user satisfaction compared to existing methods.

What Problem Does This Paper Address?

Many existing photo enhancement techniques struggle to produce aesthetically pleasing results because they either apply a uniform effect across the entire image (global enhancement) or lack an understanding of the image's content. Global methods fail to address non-uniform lighting conditions, leading to overexposed or underexposed regions. While some methods attempt local adjustments, they often lack content awareness or are computationally expensive. Furthermore, creating high-quality, locally edited datasets for training deep learning models is prohibitively expensive. This paper aims to bridge this gap by developing a method that can perform content-aware local photo enhancement efficiently, even without large, pre-annotated datasets.

Methods

The core of the proposed method is a crowd-powered local enhancement framework that utilizes an active learning-based local filter. Instead of requiring crowd workers to adjust parameters for every pixel, the system intelligently selects a small number of "key pixels" (L). For these key pixels, crowd workers locally optimize the parameters for image editing functions (brightness, saturation, contrast) using a simplified Sequential Line Search (SLS) approach, where they adjust a single slider at a time. The parameters for all other pixels are then predicted by a Gaussian Process Regression (GPR) model, trained on features derived from pixel coordinates and an illumination map.

The active learning component, specifically using the Expected Model Output Changes (EMOC) metric, is crucial for selecting the most informative key pixels. This ensures that the GPR model is trained on the most impactful data points, leading to more accurate predictions and efficient optimization. The use of an illumination map (estimated by LIME) as part of the pixel features allows the filter to be edge-aware, further enhancing its ability to make contextually appropriate adjustments.

Datasets

The paper utilizes several publicly available datasets for evaluation:

  • MEF [58], NPEA [6], LIME [11], DICM [59], VV [60], and SICE [3]: These datasets are used for evaluating the proposed local filter using an image quality assessment model.
  • SICE dataset [3]: Used specifically for full-reference image quality assessment metrics due to its availability of reference images.
  • MIT-Adobe 5K dataset [5]: Mentioned as a source for training data in other methods, but not directly used for training the proposed model due to its global enhancement focus.
  • DPED [4], LOL dataset [1], SID dataset [2], and sRGB-SID dataset [14]: These datasets are discussed in the context of limitations of existing learning-based methods, highlighting the lack of locally edited reference images.

Key Insights and Ideas

  • Crowdsourcing for Local Control: The paper demonstrates that crowdsourcing can be effectively leveraged for fine-grained, local image editing, moving beyond global adjustments.
  • Active Learning for Efficiency: By using active learning to select key pixels, the system drastically reduces the annotation burden on crowd workers, making the process more efficient and cost-effective.
  • GPR for Parameter Prediction: Gaussian Process Regression proves to be a powerful tool for predicting local image editing parameters based on a limited set of key pixels and image features.
  • Illumination Map for Edge Awareness: Incorporating an illumination map into pixel features allows the GPR model to learn edge-aware adjustments, leading to more natural and contextually appropriate enhancements.
  • SLS for Simplified Optimization: Breaking down multi-parameter optimization into a sequence of single-slider adjustments significantly simplifies the task for crowd workers, improving the quality and consistency of their input.

Key Results

  • Quantitative Evaluation: The proposed filter achieved the best scores in four out of six image quality assessment metrics (BIQME, NIQE, PSNR, SSIM) and comparable scores in the remaining two, outperforming existing local filters.
  • User Study: In a user study involving real crowd workers, the proposed method received higher ratings than all compared methods, demonstrating its ability to produce more visually pleasing results. The average voting rate for the proposed method was consistently higher across various comparisons.
  • Content-Awareness: Visualizations and analysis of parameter distributions confirmed that the method is content-aware, adapting adjustments based on image content rather than solely on illumination levels.
  • Robustness: The method showed robustness to variations in initial parameters, producing visually pleasing results across different trials.

Limitations and Future Work

  • Processing Time: The primary limitation acknowledged is the long processing time required for enhancement, which can take several hours due to the sequential nature of crowd worker involvement and task deployment.
  • Future Work: The authors suggest creating a novel dataset using their proposed method, which could facilitate the training of more efficient enhancement models. They also envision developing an enhancement model using this novel dataset to address the long processing time issue.

Summary updated

Abstract

In this study, we address local photo enhancement to improve the aesthetic quality of an input image by applying different effects to different regions. Existing photo enhancement methods are either not content-aware or not local; therefore, we propose a crowd-powered local enhancement method for content-aware local enhancement, which is achieved by asking crowd workers to locally optimize parameters for image editing functions. To make it easier to locally optimize the parameters, we propose an active learning based local filter. The parameters need to be determined at only a few key pixels selected by an active learning method, and the parameters at the other pixels are automatically predicted using a regression model. The parameters at the selected key pixels are independently optimized, breaking down the optimization problem into a sequence of single-slider adjustments. Our experiments show that the proposed filter outperforms existing filters, and our enhanced results are more visually pleasing than the results by the existing enhancement methods. Our source code and results are available at https://github.com/satoshi-kosugi/crowd-powered.
BibTeX
@article{kosugi2023crowd,
  author = {Satoshi Kosugi and Toshihiko Yamasaki},
  title = {{Crowd-Powered Photo Enhancement Featuring an Active Learning Based Local Filter}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2023},
  doi = {10.1109/tcsvt.2023.3233989},
}
Figure