TL;DR
This paper introduces two novel neural style transfer models that address common issues like color differences and underexposure in input images. By incorporating robust nonparametric distribution transfer and an adaptive exposure correction algorithm, the proposed methods significantly improve the spatial rationality and detail preservation of stylized images, leading to more aesthetically pleasing and robust results.
What Problem Does This Paper Address?
Neural style transfer (NST) aims to render a content image in the artistic style of another image. While deep learning has revolutionized this field, existing methods often struggle with two key challenges:
- Color Mismatch: When the content and style images have significantly different color dynamic ranges, the stylized output can exhibit misplaced textures and an unnatural spatial arrangement of colors. This is particularly problematic when the content image has a smaller dynamic range than the style image.
- Underexposure: Low-illumination or underexposed content images present difficulties for NST. In these images, object edges are less distinct, and color saturation is low, leading to poor content loss and unsatisfactory stylized results.
These limitations hinder the practical application of NST, especially for everyday photos taken with devices like smartphones.
Methods
The paper proposes two main contributions:
- Robust Nonparametric Distribution Transfer (RNDT): This model tackles the color mismatch issue. Before applying the standard NST pipeline, it uses a robust nonparametric kernel regression (RNKR) within an iterative distribution transform (IDT) framework. This process effectively transfers the color probability density function of the style image to the content image. This pre-processing step enhances the content image's color dynamic range and clarifies object boundaries, leading to more spatially rational stylized outputs.
- Exposure Corrected and Robust Nonparametric Distribution Transferred Neural Style Transfer (EC-RNDT): This model builds upon RNDT to address underexposed images. It incorporates an Adaptive Detail-Enhanced Multi-Scale Retinex (DEMSR) algorithm for exposure correction. DEMSR involves:
- Backlit Region Detection: Identifying underexposed areas.
- Light Restoration: Using a modified MSR algorithm with guided filters to restore natural light and detail.
- Detail Enhancement: Employing a Laplacian filter to sharpen edges and improve object stereoscopy.
- Fusion: Combining the enhanced backlit regions with normal regions. The output of DEMSR is then fed into the RNDT model for style transfer.
Datasets
Not explicitly detailed in the paper. The experiments appear to use standard image datasets for content and style images, common in NST research.
Key Insights and Ideas
- Color Distribution Matters: The paper strongly emphasizes that the difference in color dynamic ranges between content and style images is a primary cause of spatial misplacements in stylized outputs.
- Pre-processing is Key: Instead of solely relying on the NST network, the proposed methods leverage pre-processing steps (color distribution transfer and exposure correction) to significantly improve the final results.
- Robustness to Parameters: The DEMSR algorithm's backlit region detection parameters (Lī and λ) were found to have a minor impact on the final style transfer results, indicating the robustness of the overall approach.
- General Applicability: The RNDT and EC-RNDT models are designed to work as pre-processing steps for various NST methods that require a single style image, enhancing their performance without needing to retrain the entire NST architecture.
Key Results
- Qualitative Improvements: The proposed RNDT and EC-RNDT models consistently produce stylized images with more reasonable spatial structures, clearer object boundaries, and better detail preservation, especially for underexposed images, compared to baseline methods like Gatys et al., Huang et al., Li et al., Liao et al., and Johnson et al.
- Quantitative Evaluation:
- User Study: Tables 1 and 2 show that RNDT and EC-RNDT received significantly higher preference percentages (42.49% and 47.84% respectively) compared to baseline models.
- Deception Rate & FID Score: When used as pre-processing for other NST models (Gatys, Huang), the proposed methods improved deception rates (e.g., from 0.43 to 0.49 with Gatys) and lowered FID scores (e.g., from 265.3 to 262.8 with Gatys), indicating better style transfer and image quality.
- Computational Cost: RNDT has an average computational cost of 1.1 seconds, and EC-RNDT takes 4 seconds per image.
Limitations and Future Work
- Not End-to-End: The authors acknowledge that their approach is not fully end-to-end, which affects efficiency.
- Future Work: Developing an efficient end-to-end style transfer method is identified as a key area for future research.
Summary updated