Pseudo-Siamese Blind-spot Transformers for Self-Supervised Real-World Denoising

Yuhui Quan, Tianxiang Zheng, Hui JiNeurIPS 2024 · OpenReview

Real-world image denoising is a bit of a "holy grail" in computer vision. While we can easily train models to remove synthetic noise, real-world camera noise is a messy, correlated beast that varies by sensor and lighting. A new paper accepted to NeurIPS 2024 introduces SelfFormer, a novel transformer-based approach that masters denoising using only single, noisy images—no clean "ground truth" required.

TL;DR

SelfFormer combines the long-range reasoning of Transformers with the "blind-spot" technique to denoise real-world images without needing clean training pairs. By using a clever Pseudo-Siamese architecture and Directional Self-Attention, it outperforms existing self-supervised methods while remaining significantly faster than previous transformer-based denoisers.

What Problem Does This Paper Address?

Most deep-learning denoisers are "supervised," meaning they need pairs of noisy and perfectly clean images to learn. In the real world, getting these pairs is incredibly difficult—you have to take many shots of the same static scene and align them perfectly. This doesn't work for moving subjects or scientific imaging.

Self-supervised methods, specifically Blind-Spot Networks (BSNs), solve this by training a model to predict a pixel's value based only on its neighbors, excluding the pixel itself (the "blind spot"). If the noise is independent between pixels, the model can't just "cheat" by memorizing the noise; it must learn to predict the underlying clean signal.

However, real-world noise is often spatially correlated (nearby pixels share similar noise). Traditional CNN-based BSNs struggle here because their receptive fields are local. Transformers are better because they look at distant pixels, but simply "masking" pixels in a Transformer often breaks its performance and accuracy.

Methods

The authors introduce SelfFormer, which integrates the blind-spot property directly into the Transformer architecture through two main innovations:

  1. Directional Self-Attention (DSA): Instead of looking at all surrounding pixels, the DSA module uses a "half-plane" grid.
Comparison of Plain Self-Attention and the proposed Directional Self-Attention (DSA) using half-plane grids.

This creates a unidirectional flow of information. Because a token only attends to pixels in one direction (e.g., only to its left), information can never "loop back" to the original pixel, effectively creating a built-in blind spot without needing to manually mask out input data.

Grid-based multi-head implementation of the Directional Self-Attention.

To capture diverse spatial features, this grid-based approach is extended across multiple attention heads, ensuring robust feature extraction while maintaining the blind-spot constraint.

  1. Pseudo-Siamese Architecture: The model consists of two sub-networks that share weights:
  2. SelfFormer-D (Directional): Uses four branches (Up, Down, Left, Right) with DSA to enforce the blind-spot constraint during training.
  3. SelfFormer-F (Full): Uses a standard full-grid attention.

During training, these two networks engage in mutual learning. The blind-spot network (D) prevents the model from learning the identity mapping (copying noise), while the full network (F) provides a more global context. Crucially, at inference time, only the faster, more powerful SelfFormer-F is used.

Datasets

The model was evaluated on three major real-world benchmarks:

  • SIDD (Smartphone Image Denoising Dataset): Images from five different smartphones.
  • DND (Darmstadt Noise Dataset): 50 high-resolution noisy-clean pairs.
  • NIND (Natural Image Noise Dataset): Real-world images captured at various ISO levels (3200 to 6400).

Key Insights and Ideas

  • Synergy of Long-Range Interaction: The "aha moment" here is that the blind-spot technique needs long-range interactions to work well on real noise. Since noise is correlated locally, looking at a pixel 20 pixels away is much more informative for denoising than looking at its immediate neighbor.
  • The Half-Plane Trick: By restricting attention to a half-plane, the authors satisfy the mathematical requirements of a BSN while maintaining the structural integrity of the Transformer's feature maps.
  • Training vs. Inference Asymmetry: The Pseudo-Siamese setup is a clever way to have your cake and eat it too. You get the strict self-supervision constraints during training, but you don't pay the computational price of the four-branch directional architecture when actually using the model to denoise an image.

Key Results

SelfFormer sets a new state-of-the-art for "clean-image-free" denoising:

  • Accuracy: On the SIDD benchmark, it achieved a PSNR of 37.69 dB, outperforming the previous best self-supervised method (PUCA) and even some supervised models trained on synthetic noise.
  • Visual Quality: Qualitative results show that SelfFormer is much better at recovering fine textures (like fabric or text) that other BSNs tend to blur or turn into "waxy" artifacts.
  • Efficiency: Despite being a Transformer, SelfFormer-F is roughly 3x faster than previous transformer-based BSNs like LG-BPN and SS-BSN during inference, thanks to its single-path design.

Limitations and Future Work

While SelfFormer is a leap forward, the authors note a few hurdles:

  • The Supervised Gap: There is still a performance gap (about 2-3 dB) between this self-supervised method and the best fully-supervised models that have access to clean ground truth.
  • Computational Overhead: Like most Transformers, it is still more computationally expensive than simple CNNs, which might limit its use on low-power mobile hardware for real-time video denoising.
  • Future Directions: The team plans to explore making the architecture even more efficient and potentially extending it to video denoising, where temporal consistency adds another layer of complexity.

Paper Citation: Quan, Y., Zheng, T., & Ji, H. (2024). Pseudo-Siamese Blind-spot Transformers for Self-Supervised Real-World Denoising. 38th Conference on Neural Information Processing Systems (NeurIPS 2024).

Summary updated

Abstract

Real-world image denoising remains a challenge task. This paper studies self-supervised image denoising, requiring only noisy images captured in a single shot. We revamping the blind-spot technique by leveraging the transformer’s capability for long-range pixel interactions, which is crucial for effectively removing noise dependence in relating pixel–a requirement for achieving great performance for the blind-spot technique. The proposed method integrates these elements with two key innovations: a directional self-attention (DSA) module using a half-plane grid for self-attention, creating a sophisticated blind-spot structure, and a Siamese architecture with mutual learning to mitigate the performance impacts from the restricted attention grid in DSA. Experiments on benchmark datasets demonstrate that our method outperforms existing self-supervised and clean-image-free methods. This combination of blind-spot and transformer techniques provides a natural synergy for tackling real-world image denoising challenges.
BibTeX
@inproceedings{quan2024pseudo,
  author = {Yuhui Quan and Tianxiang Zheng and Hui Ji},
  title = {{Pseudo-Siamese Blind-spot Transformers for Self-Supervised Real-World Denoising}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=O3nPufVaee},
}
Figure