TL;DR
This paper introduces "Deep Preset," a novel approach to color style transfer that learns low-level image transformations rather than directly copying colors. By predicting the underlying "preset" (a set of editing parameters) used to achieve a desired look, Deep Preset enables more natural and robust color stylization, outperforming existing methods in both quantitative metrics and user studies.
What Problem Does This Paper Address?
Many users want to achieve the polished, aesthetically pleasing color styles of professionally retouched photos but lack the technical skills to use complex editing software like Adobe Lightroom. Existing image style transfer methods often struggle with this, either by directly copying colors which can lead to unnatural results when content mismatches, or by focusing on artistic styles rather than photorealistic retouching. The challenge is to transfer the essence of a color style – the underlying transformation – rather than just the colors themselves, ensuring that the stylized image looks natural and appealing, even when the content of the input image differs significantly from the reference image.
Methods
The core of the proposed method is a deep neural network named "Deep Preset," built upon a U-Net architecture. It comprises four main components:
- Encoder T: Takes both the input image (X) and a reference image (Z) to learn feature maps representing the color transformation.
- Encoder C: Extracts contextual information from the input image (X) to aid in blending.
- Linear Layers L: Processes the features from Encoder T to estimate the "preset" (P) – a vector of 69 hyper-parameters representing low-level image editing operations (like hue, saturation, contrast adjustments) that were applied to the reference image.
- Decoder G (Generator): Combines the features from T and C to synthesize the final stylized photo (Ŷ).
A key innovation is the Positive Pair-wise Loss (PPL). This loss function encourages the network to learn robust representations of color transformations by minimizing the distance between latent features of images that have been retouched with the same preset. This helps stabilize preset estimation and improve the quality of the stylized output, preventing the model from being overly sensitive to specific image content.
The training process uses a supervised approach, leveraging ground-truth data generated by scripting Adobe Lightroom to apply a large number of user-generated presets to a diverse set of images.
Datasets
- Training Data: Generated using Adobe Lightroom, comprising 601,200 photos created from 1,200 high-definition images from the Flickr2K dataset and 501 user-generated presets (with 69 settings each).
- Evaluation Data: Four subsets were prepared:
- DIV2K [21] (both 1x100x10 and 10x10x10 configurations)
- MIT-Adobe FiveK [5] (10x10x10 configuration, specifically using images retouched by "Expert C" to ensure naturalness)
- Cosplay Portraits (CP) (10x10x10 configuration, including both natural and retouched images)
Key Insights and Ideas
- Color Style as Transformation, Not Just Colors: The paper redefines color style transfer not as copying colors, but as learning the underlying low-level image transformation process that creates a desired look. This is a crucial shift that allows for more natural results.
- Predicting the "Preset": By predicting the specific editing parameters (the "preset") used to achieve a retouched look, the model gains a deeper understanding of the color transformation, leading to more controllable and interpretable results.
- The Power of Positive Pairs: The Positive Pair-wise Loss (PPL) is a clever way to enforce consistency in color style representation, making the model more robust and improving the overall quality of the stylized images. It addresses the challenge of generalizing transformations across different image contents.
- Leveraging User-Generated Presets: The use of a large dataset of real-world Lightroom presets provides a rich source of ground-truth data that reflects how actual users achieve desired color styles.
Key Results
- Quantitative Performance: Deep Preset significantly outperforms previous works (Reinhard et al., MKL, Deep Priors, FPS, WCT², PhotoNAS) across various metrics, including H-Corr, H-CHI, PSNR, and LPIPS, on all tested datasets. For example, on average, it achieves the highest PSNR (22.33 dB) and lowest LPIPS (0.1011).
- Qualitative Improvement: Visual comparisons show that Deep Preset produces more natural and aesthetically pleasing results, avoiding the unnatural colors and distortions often seen in prior methods. It successfully transfers the "vibe" or "style" of the reference without degrading image quality.
- User Study Success: A user study involving 23 volunteers showed that Deep Preset's outputs were most frequently chosen as being closest to the ground-truth, closest to the reference style, and most preferred by users, indicating its suitability for production.
Limitations and Future Work
- Preset Prediction Stability: While the PPL loss improves stability, the paper notes that predicting an exact preset can still be challenging due to the variety of possible adjustments and their dependence on image content.
- Computational Cost: While not explicitly detailed as a limitation, the use of deep neural networks implies a computational cost for training and inference.
- Future Work: The authors suggest that their work can be extended to automatically beautify photos by selecting suitable references based on perceptual measurements. Further exploration of the trade-offs between preset prediction accuracy and the quality of the stylized output is also implied. The ability to predict presets opens avenues for more interactive and controllable style transfer.
Summary updated