Learning a Deep Color Difference Metric for Photographic Images

Haoyu Chen, Zhihua Wang, Yang Yang, Qilin Sun, Kede Ma

Introduction

Measuring color difference (CD) is a fundamental task in image processing, quality assessment, and computer vision. For decades, the industry has relied on formulas such as CIELAB and CIEDE2000. While these metrics are highly effective for uniform color patches—the context in which they were originally calibrated—they often struggle when applied to complex photographic images. In real-world photography, color perception is inextricably linked to spatial context, including textures and edges. Furthermore, practical applications often involve comparing images with slight geometric misalignments, a scenario where traditional pixel-wise metrics frequently fail.

To address these challenges, this research introduces CD-Flow, a deep-learning-based color difference metric designed specifically for photographic content. By leveraging multi-scale normalizing flows, CD-Flow provides a mathematically rigorous and perceptually aligned solution that remains robust to the "messiness" of real-world image data.

Methodology: The CD-Flow Architecture

The core innovation of CD-Flow is the use of a multi-scale autoregressive normalizing flow (a variant of the RealNVP architecture). Unlike standard convolutional neural networks that may lose information through non-invertible operations, normalizing flows are bijective. This ensures that the transformation into the latent space is a proper metric, satisfying essential mathematical properties such as symmetry and the triangle inequality.

Multi-Scale Processing

The model processes images across six distinct scales. At each scale, a "squeezing" operation is employed to trade spatial resolution for channel depth. This architecture mimics the human visual cortex's integrated processing of color and form. By analyzing the image at multiple resolutions, the model can capture both fine-grained textural differences and coarse-scale color shifts.

Hybrid Loss Function

The training process utilizes a hybrid loss function composed of two parts:

  1. CD Loss: This term aligns the model's predictions with human subjective ratings from the SPCD dataset.
  2. Negative Log-Likelihood (NLL) Loss: This acts as a regularizer, ensuring the learned transformation remains a stable and valid probability distribution. This dual approach prevents the model from overfitting to specific artifacts while maintaining high sensitivity to perceptual differences.

Experimental Setup and Datasets

The researchers evaluated CD-Flow using the Smartphone Perceptual Color Difference (SPCD) dataset, currently the largest and most comprehensive dataset of its kind. SPCD contains over 15,000 images and 30,000 image pairs, each accompanied by approximately 20 human ratings.

To ensure the model's generalizability, it was also tested against:

  • COM Dataset: A collection of traditional color patch datasets (including BFD-P, Leeds, and Witt).
  • TID2013 (Subset): Used to evaluate performance against specific distortions like quantization noise and chromatic aberration.

Key Insights

The Inseparability of Color and Form

A primary finding of the study is that color perception in photographs cannot be isolated from spatial structure. CD-Flow's success is largely attributed to its ability to account for textures and boundaries. Analysis of the model's internal "local CD maps" reveals responses to edges similar to the double-opponent cells found in the primary visual cortex.

Robustness to Misalignment

Traditional metrics are often hypersensitive to pixel-level shifts. CD-Flow overcomes this by automatically learning to prioritize coarse-scale representations when images are misaligned. This allows the metric to ignore "pixel jitter" caused by camera movement or parallax while still accurately capturing the underlying shift in color appearance.

The Advantage of Bijectivity

By enforcing invertibility via normalizing flows, the authors ensure that every change in the input RGB space results in a corresponding change in the latent feature space. This avoids the "black box" problem common in deep learning, where subtle but perceptually significant color shifts might otherwise be ignored by the network.

Results and Performance

CD-Flow demonstrates a significant performance leap over 15 existing metrics, including CIEDE2000, SSIM, and LPIPS. On the SPCD benchmark, CD-Flow achieved a Pearson Correlation (PLCC) of 0.871, markedly higher than the 0.726 achieved by CIEDE2000.

Furthermore, in tests involving non-perfectly aligned image pairs, CD-Flow maintained high accuracy, whereas the error rates (measured by STRESS) for traditional pixel-wise metrics increased dramatically. Notably, despite being trained on complex photographs, the model remained competitive on traditional uniform color patch datasets, indicating a robust understanding of color science that transcends specific image types.

Conclusion and Future Directions

CD-Flow represents a significant advancement in the field of color science, bridging the gap between traditional colorimetry and modern deep learning. While the model is computationally more intensive than simple formulas due to its multi-scale flow architecture, its accuracy and robustness make it a powerful tool for high-end image processing pipelines.

Future research may focus on optimizing the architecture for real-time applications, such as in-camera processing, or extending the framework to handle temporal consistency in video sequences.

Summary updated

Abstract

Most well-established and widely used color difference (CD) metrics are handcrafted and subject-calibrated against uniformly colored patches, which do not generalize well to photographic images characterized by natural scene complexities. Constructing CD formulae for photographic images is still an active research topic in imaging/illumination, vision science, and color science communities. In this paper, we aim to learn a deep CD metric for photographic images with four desirable properties. First, it well aligns with the observations in vision science that color and form are linked inextricably in visual cortical processing. Second, it is a proper metric in the mathematical sense. Third, it computes accurate CDs between photographic images, differing mainly in color appearances. Fourth, it is robust to mild geometric distortions (e.g., translation or due to parallax), which are often present in photographic images of the same scene captured by different digital cameras. We show that all these properties can be satisfied at once by learning a multi-scale autoregressive normalizing flow for feature transform, followed by the Euclidean distance which is linearly proportional to the human perceptual CD. Quantitative and qualitative experiments on the large-scale SPCD dataset demonstrate the promise of the learned CD metric.
BibTeX
@inproceedings{chen2023learning,
  author = {Haoyu Chen and Zhihua Wang and Yang Yang and Qilin Sun and Kede Ma},
  title = {{Learning a Deep Color Difference Metric for Photographic Images}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure