Abstract
Camera Image Signal Processing (ISP) pipelines, including deep learning trained versions, can get appealing results in different image signal processing tasks. However, most if not all of these methods tend to apply a single filter that is homogeneous over the entire image. This is also particularly true when an encoder-decoder type deep architecture is trained for the task. However, it is natural to view a camera image as heterogeneous, as the color intensity and the artificial noise are distributed vastly different, even across the two dimensional domain of a single image. Varied Moire ringing, motion-blur, color-bleaching or lens based projection distortions can all potentially lead to a heterogeneous image artifact filtering problem. In this paper, we present a specific patch-based, local subspace deep neural network that improves Camera ISP to be robust to heterogeneous artifacts (especially image denoising). We call our three-fold deep trained model the Patch Subspace Learning Autoencoder (PSL-AE). PSL-AE does not necessarily assume uniform image distortion levels nor repeated nor similar artifact types within the image. Rather, PSL-AE first diagnostically encodes patches extracted from noisy and clean image pairs, with different artifact type and distortion levels, by contrastive learning. Then, each image's patches are encoded into soft-clusters in their appropriate latent sub-space, using a prior mixture model. Lastly, the decoders of the PSL-AE are also trained in an unsupervised manner customized for the image patches in each soft-cluster. Our experimental results demonstrates the flexibility and performance that one can achieve through improved heterogeneous filtering, both from synthesized artifacts but also realistic SIDD image pairs.
Introduction
Standard Camera Image Signal Processing (ISP) pipelines typically apply filters uniformly across an entire image frame. However, real-world images are inherently heterogeneous; factors such as lens projection distortions, varying color intensities, and sensor limitations create non-uniform noise profiles. For instance, a single capture might exhibit motion blur in one region and heavy sensor grain or Moire ringing in another.
Existing deep learning models often rely on the assumption of a "universal artifact," such as simple Gaussian noise. When applied to the complex, non-universal noise characteristic of smartphone photography, these "one-size-fits-all" models often fall short. This paper introduces the Patch Subspace Learning Autoencoder (PSL-AE), a framework designed to adapt to local noise characteristics by treating images as a collection of patches rather than a monolithic block.
The PSL-AE Framework
The PSL-AE architecture moves away from uniform filtering toward a local, patch-based approach. The process is divided into three primary stages:
- Contrastive Pre-training: To ensure the encoder focuses on the underlying content rather than the noise, the model employs a contrastive learning strategy. By maximizing the agreement between a noisy patch and its clean counterpart in a latent space, the encoder learns robust "identity" features.
- Latent Subspace Clustering: The encoder maps patches into a latent space where they are grouped into soft-clusters using a prior mixture model. This allows the system to categorize patches based on their specific noise levels or patterns without requiring explicit labels.
- Multi-Decoder Reconstruction: Instead of a single general-purpose decoder, PSL-AE utilizes a "committee" of specialized decoders. Each decoder is optimized to reconstruct patches from a specific latent subspace. During inference, patches are routed to the decoder that best matches their identified noise profile.
To prevent "grid" artifacts at patch boundaries, the framework uses overlapping patches and applies linear interpolation to ensure smooth transitions in the final assembled image.
Experimental Results
The model was evaluated on both synthesized and real-world datasets, including a synthesized version of CelebA with mixed noise types and the high-standard Smartphone Image Denoising Dataset (SIDD).
Performance and Efficiency
On the synthesized CelebA dataset, PSL-AE achieved a 31.74 PSNR, significantly outperforming standard convolutional autoencoders (31.02) and classical methods like BM3D (25.13). On the SIDD dataset, it reached 33.81 PSNR.
Beyond accuracy, PSL-AE is highly efficient. With only 12.58G FLOPs, it is approximately 12 times more efficient than a standard UNet (116.38G FLOPs), making it a strong candidate for on-device mobile ISP applications.
Conclusion
PSL-AE demonstrates that a committee of specialized, simple decoders can outperform a single complex model when dealing with diverse, non-uniform artifacts. By leveraging contrastive learning and unsupervised latent clustering, the model naturally discovers how to partition and address different types of image corruption. Future work will explore extending this subspace approach to other ISP tasks like de-mosaicing and motion blur removal.
Summary updated