Stochastic Window Transformer for Image Restoration

Jie Xiao, Xueyang Fu, Feng Wu, Zheng-Jun ZhaNeurIPS 2022 · OpenReview

TL;DR

The Stochastic Window Transformer (Stoformer) addresses a major flaw in current vision transformers: the loss of translation invariance and local connectivity caused by fixed window partitions. By introducing random window shifts during training and an efficient "layer expectation propagation" algorithm for testing, Stoformer achieves state-of-the-art results in image deraining, denoising, and deblurring while eliminating unsightly blocking artifacts.

The Problem: Broken Locality and Invariance

In the domain of image restoration (such as removing noise, blur, or rain), two properties are paramount:

  1. Locality: Nearby pixels are highly correlated. A model must look at a pixel's immediate neighbors to restore it accurately.
  2. Translation Invariance: A restoration "filter" should work identically regardless of where the degradation is located in the image.

Convolutional Neural Networks (CNNs) handle these naturally through sliding kernels. However, Transformers—despite their powerful representation capabilities—struggle here. To process high-resolution images efficiently, Transformers often split images into fixed, non-overlapping windows (e.g., the Swin Transformer).

The authors highlight two critical failures of this fixed-window approach:

  • Broken Locality: Pixels sitting on opposite sides of a window boundary cannot "talk" to each other, leading to a massive loss of local context.
  • Broken Translation Invariance: The model becomes biased toward the specific grid it was trained on. If a blurry spot shifts by just a few pixels, the restoration quality can fluctuate wildly, often leaving behind visible "grid" or "blocking" artifacts.

Methods: The Stochastic Window Strategy

To overcome these limitations, the authors propose the Stochastic Window Strategy, which consists of two main components integrated into a hierarchical architecture.

Stoformer Architecture

1. Stochastic Window Partition (Training)

Instead of using a fixed grid or a simple alternating shift, the authors treat the window offset as a random variable. During training, for every layer, the window partition is shifted by a random amount $(\xi_h, \xi_w)$. This ensures that every pixel has an equal chance of being in the center, edge, or corner of a window, forcing the model to learn features that are independent of the grid's position.

2. Layer Expectation Propagation (Testing)

Ideally, at test time, one would run the image through every possible window shift and average the results. However, for a deep network, this would require an exponential number of forward passes—far too slow for practical use.

To solve this, the authors designed a Layer Expectation Propagation (LEP) algorithm. It approximates the "average" of all possible shifts in a single forward pass by calculating the expectation of the signal at each layer. This gives the model the robustness of a sliding window with the speed of a fixed-window Transformer.

Key Insights and Ideas

  • The "Aha" Moment on Locality: The authors quantitatively demonstrated the severe impact of fixed boundaries. In a fixed $8 \times 8$ window, nearly 40% of local relationships (pixels within a small radius) are lost because they fall into different windows.
Captured Locality vs. Radius

As shown above, Stoformer's stochastic approach recovers 100% of these relationships up to the window size.

  • Implicit Ensemble: Training with stochastic shifts acts as an "implicit model ensemble." Effectively, the model is trained on millions of different window configurations, which significantly boosts its ability to generalize to new images.
  • Efficiency vs. Quality: While a true "sliding window" Transformer would be the perfect theoretical solution, it is computationally ruinous. Stoformer provides an elegant "middle way"—achieving sliding-window-like invariance using the memory footprint of a fixed-window model.

Key Results

Stoformer was evaluated across three major restoration tasks (Deraining, Denoising, and Deblurring) and consistently outperformed previous state-of-the-art models like Uformer, SwinIR, and Restormer:

  • Deraining: On the SPA-Data benchmark, Stoformer achieved a PSNR of 48.97 dB, a significant jump over Uformer’s 47.84 dB.
  • Denoising: Stoformer set new high-water marks across almost all noise levels. For example, on the Urban100 dataset (color, $\sigma=50$), it reached 30.06 dB, outperforming Restormer (30.02 dB) and significantly beating older CNN-based methods.
  • Visual Quality: Qualitatively, the results are much cleaner. In a "Toy Experiment" analyzing error distributions, the authors showed that while fixed-window models produce a "bumpy" error distribution tied to the grid, Stoformer’s error is flat and uniform—proving it has successfully mastered translation invariance.

Limitations and Future Work

  • Architecture Scope: The current study focuses primarily on a U-shaped (hierarchical) architecture. Future work should validate this stochastic strategy on other Transformer shapes, such as isotropic or multi-stage designs.
  • Task Expansion: While highly effective for restoration, the authors plan to extend this logic to other low-level vision tasks like Super-Resolution, JPEG deblocking, and Video Restoration, where temporal translation invariance is also a factor.
  • Approximation Gap: While Layer Expectation Propagation is efficient, it remains an approximation of the true expectation. Exploring even more precise ways to marginalize the stochastic shifts without adding compute time remains an open research area.

Summary updated

Abstract

Thanks to the powerful representation capabilities, transformers have made impressive progress in image restoration. However, existing transformers-based methods do not carefully consider the particularities of image restoration. In general, image restoration requires that an ideal approach should be translation-invariant to the degradation, i.e., the undesirable degradation should be removed irrespective of its position within the image. Furthermore, the local relationships also play a vital role, which should be faithfully exploited for recovering clean images. Nevertheless, most transformers either adopt local attention with the fixed local window strategy or global attention, which unfortunately breaks the translation invariance and causes huge loss of local relationships. To address these issues, we propose an elegant stochastic window strategy for transformers. Specifically, we first introduce the window partition with stochastic shift to replace the original fixed window partition for training. Then, we design a new layer expectation propagation algorithm to efficiently approximate the expectation of the induced stochastic transformer for testing. Our stochastic window transformer not only enjoys powerful representation but also maintains the desired property of translation invariance and locality. Experiments validate the stochastic window strategy consistently improves performance on various image restoration tasks (deraining, denoising and deblurring) by significant margins. The code is available at https://github.com/jiexiaou/Stoformer.
BibTeX
@inproceedings{xiao2022stochastic,
  author = {Jie Xiao and Xueyang Fu and Feng Wu and Zheng-Jun Zha},
  title = {{Stochastic Window Transformer for Image Restoration}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=5gpBVVfmcw},
}

Loading related content...

Figure