HomoFormer: Homogenized Transformer for Image Shadow Removal

Jie Xiao, Xueyang Fu, Yurui Zhu, Dong Li, Jie Huang, Kai Zhu, Zheng-Jun Zha

Abstract

Image shadow removal is a spatially heterogeneous problem where degradation levels vary significantly across an image. Conventional deep learning models relying on weight sharing often struggle to specialize for both shadowed and non-shadowed regions simultaneously. We introduce HomoFormer, a novel architecture that employs a "homogenization" strategy. By applying a stochastic random shuffle operation, HomoFormer distributes non-uniform shadow degradation across the entire image, allowing standard local window Transformers to operate on a homogenized data distribution. To preserve spatial context, we integrate a Structure Modeling MLP (SMLP) that restores local relationships. Our approach achieves state-of-the-art results on major benchmarks, including ISTD+ and SRD.

Introduction: The Challenge of Spatial Heterogeneity

Shadow removal requires handling diverse patterns: some regions are totally shadowed, while others are partially shadowed or clear. Most dominant architectures, including CNNs and window-based Transformers (e.g., Swin Transformer), utilize weight sharing. This forces the model to find an unsatisfactory compromise between parameters suited for dark shadow cores and those for soft edges. While global Transformers could theoretically address this, their quadratic computational complexity makes them impractical for high-resolution restoration.

Concept of homogenization via random shuffling.

Methodology: The Homogenized Transformer

HomoFormer addresses non-uniformity through a three-step homogenization process within its core blocks:

  1. Random Shuffle ($S$): Pixel features are stochastically rearranged across the spatial domain. This spreads concentrated shadow patches thinly and evenly across the entire feature map.
  2. Local Self-Attention: The homogenized features are processed by a standard local window-based self-attention layer. Because self-attention is permutation equivariant, it can effectively process shuffled tokens without losing information. This mechanism allows local windows to capture sparse global interactions, as each window now contains a random sample of pixels from the whole image.
  3. Inverse Shuffle ($I_S$): Features are returned to their original spatial positions to restore the image's semantic structure.

To compensate for the loss of local structural information during shuffling, the authors replace the standard Feed-Forward Network (FFN) with a Structure Modeling MLP (SMLP). This module utilizes depth-wise convolutions to re-learn spatial relationships that the shuffling step temporarily discards.

The HomoFormer architecture and HomoBlock details.

Experimental Results

We evaluated HomoFormer on two primary benchmarks:

  • Adjusted ISTD (ISTD+): HomoFormer achieved a Mean Absolute Error (MAE) of 2.6 for the whole image, outperforming previous leaders like ShadowDiffusion (2.7) and BMNet (3.0).
  • SRD Dataset: The model reached a PSNR of 35.37 dB, a significant improvement over the previous state-of-the-art, ShadowFormer (32.90 dB).

Qualitative analysis indicates that HomoFormer produces cleaner results with fewer boundary artifacts compared to existing methods. Furthermore, the stochastic nature of the shuffle allows for uncertainty estimation; the variance across multiple runs correlates with potential model errors, providing an implicit confidence map.

Conclusion and Future Work

HomoFormer demonstrates that "homogenizing" data to fit simpler models can be more effective than building increasingly complex adaptive architectures. While the current implementation uses Monte Carlo averaging (HomoFormer+) to mitigate inference stochasticity, the underlying philosophy of homogenization holds promise for other spatially heterogeneous tasks, such as image inpainting and non-uniform deblurring.

Summary updated

Abstract

The spatial non-uniformity and diverse patterns of shadow degradation conflict with the weight sharing manner of dominant models, which may lead to an unsatisfactory compromise. To tackle with this issue, we present a novel strategy from the view of shadow transformation in this paper: directly homogenizing the spatial distribution of shadow degradation. Our key design is the random shuffle operation and its corresponding inverse operation. Specifically, random shuffle operation stochastically rearranges the pixels across spatial space and the inverse operation recovers the original order. After randomly shuffling, the shadow diffuses in the whole image and the degradation appears in a homogenized way, which can be effectively processed by the local self-attention layer. Moreover, we further devise a new feed forward network with position modeling to exploit image structural information. Based on these elements, we construct the final local window based transformer named HomoFormer for image shadow removal. Our HomoFormer can enjoy the linear complexity of local transformers while bypassing challenges of non-uniformity and diversity of shadow. Extensive experiments are conducted to verify the superiority of our HomoFormer across public datasets. Code is available at https://github.com/jiexiaou/HomoFormer.
BibTeX
@inproceedings{xiao2024homoformer,
  author = {Jie Xiao and Xueyang Fu and Yurui Zhu and Dong Li and Jie Huang and Kai Zhu and Zheng-Jun Zha},
  title = {{HomoFormer: Homogenized Transformer for Image Shadow Removal}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02420},
}
Figure