Design Principles of Multi-Scale J-Invariant Networks for Self-Supervised Image Denoising

Hayeong Yu, Seungjae Han, Young-Gyu YoonWACV 2025

TL;DR

This paper introduces a new set of flexible design principles for self-supervised image denoising networks, allowing for the incorporation of diverse architectural elements like U-Nets. They demonstrate this with U-BSN, a U-Net-based denoising model that achieves state-of-the-art performance with significantly reduced computational cost, and introduce Randomized PD to further enhance robustness on real-world noisy images.

What Problem Does This Paper Address?

Image denoising is a critical task, but obtaining paired noisy and clean images for supervised training is often impractical. Self-supervised learning offers a solution by learning from noisy images alone, typically by enforcing a "J-invariance" property. This property ensures that the denoised output for a pixel is independent of its direct input, preventing the network from simply learning to copy the input. However, traditional methods for enforcing J-invariance, particularly those using blind spot networks (BSNs), impose strict constraints on network architecture. These constraints often exclude efficient multi-scale designs like U-Nets and lead to complex, computationally expensive models. This paper aims to relax these constraints by proposing generalized design principles for multi-scale J-invariant networks, enabling more flexible and efficient architectures.

Methods

The core contribution lies in the generalized design principles for multi-scale J-invariant networks. Instead of requiring J-invariance throughout the entire network, the authors propose rules that allow for its flexible infusion. They analyze two key components of multi-scale architectures: residual connections and channel-wise concatenation, and how they interact with J-invariance.

The paper introduces two main technical innovations:

  • U-BSN: This is a novel multi-scale J-invariant network built upon the U-Net architecture. It incorporates blind spot convolution layers within each skip connection. This placement is strategic, aiming to reduce computational costs by applying the constraint before upsampling stages. The architecture is designed to leverage the U-Net's inherent multi-scale processing capabilities while adhering to the relaxed J-invariance principles.
  • Randomized PD (Pixel-Shuffle Downsampling): To address the challenge of spatially correlated noise in real-world images, the authors propose an enhancement to pixel-shuffle downsampling. Randomized PD involves using random PD strides, intrapatch permutation (shuffling pixels within patches), and patchwise rotation. This randomization effectively disrupts structured noise patterns without significantly increasing computational complexity, thereby improving denoising robustness.

Datasets

The paper uses the following datasets for training and evaluation:

  • Smartphone Image Denoising Dataset (SIDD): Used for both training and testing. The SIDD Medium dataset was used for training (320 noisy/clean pairs), and the SIDD validation and benchmark sets (1280 patches of size 256x256 each) were used for validation and testing. This dataset contains real-world noisy images with spatially correlated noise.
  • Darmstadt Noise Dataset (DND): Used for evaluation. This dataset consists of 50 noisy images without corresponding clean images.

Key Insights and Ideas

  • J-invariance doesn't need to be enforced everywhere: The most significant insight is challenging the dogma that J-invariance must be maintained at every single step of a denoising network. By proposing generalized rules, the authors unlock the potential to use more efficient and flexible architectures like U-Nets.
  • Strategic placement of blind spots matters: Placing blind spot convolution layers in skip connections of a U-Net is shown to be an effective way to achieve J-invariance in a multi-scale context without disrupting the overall flow or requiring overly complex designs.
  • Randomization is key for real-world noise: Spatially correlated noise is a major hurdle. The proposed Randomized PD method demonstrates that introducing controlled randomness (in strides, permutations, and rotations) is a powerful technique to break these noise structures, leading to better performance on challenging real-world images.
  • Efficiency gains are substantial: By combining a flexible architecture (U-Net) with effective noise handling (Randomized PD), the proposed U-BSN achieves competitive or superior denoising performance with significantly lower computational cost (fewer MACs, parameters, and faster training/inference) compared to existing state-of-the-art methods.

Key Results

  • Performance: U-BSN, especially when combined with Randomized PD, achieves state-of-the-art PSNR and SSIM scores on both SIDD and DND datasets, outperforming methods like AP-BSN, LG-BPN, and MM-BSN. For instance, on the DND benchmark, U-BSN achieves 38.56 dB / 0.942 SSIM, which is the highest among the compared methods.
  • Efficiency: U-BSN demonstrates significant efficiency improvements. It achieves the shortest training time (1.63x faster) and lowest memory usage (2.36x lower) compared to MM-BSN. It also boasts the lowest computational complexity (5.10x fewer MACs and 2.08x fewer parameters) while maintaining competitive performance.
  • Randomized PD Impact: Ablation studies show that using random PD strides (especially including stride 2) improves performance. The full Randomized PD (random strides, intrapatch permutation, and patchwise rotation) yields the highest PSNR and SSIM values.

Limitations and Future Work

  • Limitations: While the paper demonstrates the effectiveness of its principles, the specific implementation of U-BSN is a U-Net variant. It's possible that other multi-scale architectures might benefit differently or require further adaptation. The paper also notes that the effectiveness of Randomized PD is demonstrated through comparative experiments, implying that a deeper theoretical analysis of its impact on various noise types could be beneficial.
  • Future Work: The authors suggest that their generalized design principles offer a general strategy for creating J-invariant networks, implying that this framework can be applied to develop various other denoising architectures. Further exploration of different network backbones and more complex noise patterns could be future directions. The paper also implicitly suggests that further research into optimizing the randomization parameters in Randomized PD could yield even better results.

Summary updated

BibTeX
@inproceedings{yu2025design,
  author = {Hayeong Yu and Seungjae Han and Young-Gyu Yoon},
  title = {{Design Principles of Multi-Scale J-Invariant Networks for Self-Supervised Image Denoising}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure