Deep Contrastive Patch-Based Subspace Learning for Camera Image Signal Processing

Yunhao Yang, Yi Wang, Chandrajit BajajarXiv 2021 · arXiv

Abstract

Camera Image Signal Processing (ISP) pipelines can get appealing results in different image signal processing tasks. Nonetheless, the majority of these methods, including those employing an encoder-decoder deep architecture for the task, typically utilize a uniform filter applied consistently across the entire image. However, it is natural to view a camera image as heterogeneous, as the color intensity and the artificial noise are distributed vastly differently, even across the two-dimensional domain of a single image. Varied Moire ringing, motion blur, color-bleaching, or lens-based projection distortions can all potentially lead to a heterogeneous image artifact filtering problem. In this paper, we present a specific patch-based, local subspace deep neural network that improves Camera ISP to be robust to heterogeneous artifacts (especially image denoising). We call our three-fold deep-trained model the Patch Subspace Learning Autoencoder (PSL-AE). The PSL-AE model does not make assumptions regarding uniform levels of image distortion. Instead, it first encodes patches extracted from noisy and clean image pairs, with different artifact types or distortion levels, by contrastive learning. Then, the patches of each image are encoded into corresponding soft clusters within their suitable latent sub-space, utilizing a prior mixture model. Furthermore, the decoders undergo training in an unsupervised manner, specifically trained for the image patches present in each cluster. The experiments highlight the adaptability and efficacy through enhanced heterogeneous filtering, both from synthesized artifacts but also realistic SIDD image pairs.

The Challenge of Heterogeneous Noise

Traditional Camera Image Signal Processing (ISP) pipelines and many modern deep learning models share a common flaw: they assume noise and artifacts are distributed uniformly across an image. In real-world scenarios, image degradation is inherently heterogeneous. Factors such as sensor heat, lens-based projection distortions, and varying color intensities mean that a patch in the shadows might suffer from vastly different noise characteristics than a patch in the highlights. Applying a single global filter often leads to over-smoothing in some areas and under-denoising in others. The PSL-AE framework addresses this by adapting its denoising strategy locally based on the specific noise profile present in different image regions.

The PSL-AE Framework

The authors propose a three-stage training pipeline for the Patch Subspace Learning Autoencoder (PSL-AE):

  1. Patch-Based Contrastive Pre-training: The encoder is first trained to recognize that a noisy patch and its clean counterpart represent the same underlying content. This ensures the model learns robust feature representations invariant to specific noise masks.
  2. Latent Subspace Clustering: The encoder maps patches into a latent space where they are categorized into "soft clusters" or subspaces using a prior mixture model. This groups patches sharing similar noise levels or artifact types without requiring explicit labels.
  3. Multi-Decoder Denoising: The architecture employs $k$ specialized decoders. During inference, a patch is routed to the decoder corresponding to its identified subspace. These decoders act as "local experts" trained specifically for the distortions found in their assigned cluster.
The PSL-AE inference pipeline routes patches to specialized decoders based on latent subspace clustering.

To prevent visible seams where patches meet, the system uses an overlapping patch strategy with linear interpolation to smooth transitions during final assembly.

Experimental Results

The model was evaluated on two primary datasets: Synthesized CelebA, with manually injected heterogeneous noise (Gaussian, Poisson, and Salt & Pepper), and the SIDD (Smartphone Image Denoising Dataset), which contains complex, real-world artifacts.

Key Performance Metrics

  • Synthetic Heterogeneous Noise: On the CelebA dataset, PSL-AE achieved a PSNR of 31.74, significantly outperforming standard models like BM3D (25.13) and single-decoder architectures (31.02).
  • Real-World Performance: On the challenging SIDD dataset, PSL-AE reached a PSNR of 33.81, surpassing established deep learning baselines like DnCNN (30.71) and CBDNet (33.28).

Efficiency and Specialization

A key insight from the ablation studies is the power of specialization. As the number of decoders increases, the PSNR improves. Crucially, even when controlling for computational complexity (Total FLOPs), the multi-decoder approach consistently outperformed single-decoder models. This proves that the architectural "division of labor" is more effective than simply increasing the size of a monolithic network.

PSNR performance relative to the number of filters (decoders), demonstrating the benefit of specialization even under equal computational constraints.

Conclusion

The PSL-AE model demonstrates that treating images as a collection of heterogeneous patches rather than a uniform whole leads to superior denoising performance. By combining contrastive learning with specialized local experts, the model effectively handles the non-uniform noise distributions typical of real-world camera sensors. Future work will focus on preserving ultra-fine details and optimizing the routing mechanism for high-resolution 4K and 8K imagery.

Summary updated

Abstract

Camera Image Signal Processing(ISP) pipelines can get appealing results in different image signal processing tasks. Nonetheless, the majority of these methods, including those employing an encoder-decoder deep architecture for the task, typically utilize a uniform filter applied consistently across the entire image. However, it is natural to view a camera image as heterogeneous, as the color intensity and the artificial noise are distributed vastly differently, even across the two-dimensional domain of a single image. Varied Moire ringing, motion blur, color-bleaching, or lens-based projection distortions can all potentially lead to a heterogeneous image artifact filtering problem. In this paper, we present a specific patch-based, local subspace deep neural network that improves Camera ISP to be robust to heterogeneous artifacts (especially image denoising). We call our three-fold deep-trained model the Patch Subspace Learning Autoencoder (PSL-AE)<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup><sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>Oursource code is publiclyavailable at https://github.com/CVC-Lab/Patch-Subspace-Learning-Autoencodd.. The PSL-AE model does not make assumptions regarding uniform levels of image distortion. Instead, it first encodes patches extracted from noisy a nd clean image pairs, with different artifact types or distortion levels, by contrastive learning. Then, the patches of each image are encoded into corresponding soft clusters within their suitable latent sub-space, utilizing a prior mixture model. Furthermore, the decoders undergo training in an unsupervised manner, specifically trained for the image patches present in each cluster. The experiments highlight the adaptability and efficacy through enhanced heterogeneous filtering, both from synthesized artifacts but also realistic SIDD image pairs.
BibTeX
@misc{yang2021deep,
  author = {Yunhao Yang and Yi Wang and Chandrajit Bajaj},
  title = {{Deep Contrastive Patch-Based Subspace Learning for Camera Image Signal Processing}},
  year = {2021},
  eprint = {2104.00253},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.00253},
}
Figure