A Coarse-to-Fine Framework for Learned Color Enhancement with Non-Local Attention

Chaowei Shan, Zhizheng Zhang, Zhibo Chen

TL;DR

Researchers from the University of Science and Technology of China (USTC) have developed a two-stage "coarse-to-fine" deep learning framework that balances global color harmony with local detail enhancement. By combining a global channel-wise mapping network with a pixel-wise refinement network featuring non-local attention, the model achieves state-of-the-art results in automatic photo retouching.

What Problem Does This Paper Address?

Automatic color enhancement is a balancing act. If a model applies a purely global adjustment (like a single filter), it maintains a "harmonious" look but fails to fix local issues like shadows or specific object colors. Conversely, if a model adjusts every pixel independently, it can create "artifacts"—weird splotches or inconsistent colors—because the model loses track of the overall "vibe" or context of the image.

Current deep learning methods often struggle to handle both global style and local detail simultaneously. The authors aim to bridge this gap by mimicking the workflow of a professional editor: first setting the overall tone, then refining specific areas.

Methods

The authors propose a cascaded architecture consisting of two primary components:

  1. CENet (Channel-wise Enhancement Network): This network handles the "coarse" step. Using a ResNet50 backbone, it predicts a 12-dimensional vector representing a $3 \times 3$ weight matrix and a $3 \times 1$ bias vector. This defines a global linear mapping for the RGB channels. Because this mapping is applied to the whole image, it ensures global color consistency without introducing spatial artifacts.
  2. PRNet (Pixel-wise Refinement Network): This handles the "fine" step. It takes the output of the CENet and learns a "residual" image—essentially a map of local corrections to be added to the coarse result.
  3. Non-Local Attention: The "secret sauce" in PRNet is a non-local attention block. Unlike standard convolutions that only look at nearby pixels, this block allows the network to capture long-range dependencies. This helps maintain color consistency across distant parts of the image (e.g., ensuring the sky on the left matches the sky on the right).

The entire system is trained using Mean Squared Error (MSE) loss against professionally retouched ground-truth images.

Datasets

The study uses the MIT-Adobe FiveK dataset, the gold standard for this field. It contains 5,000 raw images, each retouched by five different professional photographers.

  • Training/Testing: Following standard practice, the authors used the retouches from "Photographer C" as the target. They trained on 4,750 pairs and tested on the "RANDOM250" subset.
  • Preprocessing: Images were resized to 500 pixels on the longer edge for consistency during training.

Key Insights and Ideas

  • Decoupling Global and Local: By forcing the first network (CENet) to only output 12 parameters, the authors prevent it from over-fitting to local details, ensuring it focuses purely on the "global style."
  • Residual Learning for Refinement: Instead of asking PRNet to generate a new image from scratch, it only predicts the difference (residual) needed to fix the coarse image. This makes the learning task significantly easier and more stable.
  • Attention as a Consistency Tool: The inclusion of non-local attention is a clever way to solve the "local artifact" problem. It gives the pixel-wise network a "global view," preventing it from making local adjustments that clash with the rest of the image.

Key Results

The framework was evaluated using $L^2$ error in the CIELab color space (where lower is better), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM).

  • Ablation Study: The full model (CE+PRNL) outperformed versions that used only global mapping or only local refinement. It achieved an $L^2$ error of 9.10, compared to 10.32 for global-only (CE) and 10.93 for local-only (PR).
  • Benchmark Comparison: The model outperformed several established methods, including:
  • DeepRL-based [16]: $L^2$ error of 10.99.
  • DPE [3]: $L^2$ error of 9.93.
  • Our Model: $L^2$ error of 9.10 and PSNR of 24.19 dB.
  • Qualitative Quality: Visual results show that the model avoids the blurring and "patchy" artifacts seen in purely pixel-wise methods while achieving much closer color matches to professional ground truth than purely global methods.

Limitations and Future Work

While the paper demonstrates strong results on 500px images, the authors note in the introduction that generating high-resolution retouched images with realistic effects remains a significant challenge for the field. The computational cost of non-local attention blocks can scale poorly with very high resolutions, suggesting a need for more efficient attention mechanisms or multi-scale approaches in future iterations. Additionally, the model currently relies on paired data (raw vs. retouched), which is expensive to collect. Exploring unsupervised or semi-supervised versions of this coarse-to-fine approach could be a promising next step.

Summary updated

Abstract

Automatic color enhancement are aimed to automaticly and adaptively adjust photos to expected styles and tones. For current learned methods in this field, global harmonious perception and local details are hard to be well-considered in a single model simultaneously. To address this problem, we propose a coarse-to-fine framework with non-local attention for color enhancement in this paper. Within our framework, we propose to divide enhancement process into channel-wise enhancement and pixel-wise refinement performed by two cascaded Convolutional Neural Networks (CNNs). In channel-wise enhancement, our model predicts a global linear mapping for RGB channels of input images to perform global style adjustment. In pixel-wise refinement, we learn a refining mapping using residual learning for local adjustment. Further, we adopt a non-local attention block to capture the long-range dependencies from global information for subsequent fine-grained local refinement. We evaluate our proposed framework on the commonly using benchmark and conduct sufficient experiments to demonstrate each technical component within it.
BibTeX
@inproceedings{shan2019coarse,
  author = {Chaowei Shan and Zhizheng Zhang and Zhibo Chen},
  title = {{A Coarse-to-Fine Framework for Learned Color Enhancement with Non-Local Attention}},
  booktitle = {ICIP},
  year = {2019},
  url = {https://openreview.net/forum?id=NUKzdynIzDc},
}

Loading related content...

Figure