Towards Scale-Free Rain Streak Removal via Self-Supervised Fractal Band Learning

Wenhan Yang, Shiqi Wang, Dejia Xu, Xiaodong Wang, Jiaying Liu

Abstract

Deep learning models for single-image deraining often struggle with scale variance, failing to generalize to rain streaks significantly larger or smaller than those in the training data. This paper presents Fractal Band Learning (FBL), a recursive architecture designed to capture multi-scale dependencies through hierarchical refinement. Coupled with a novel cross-scale self-supervision constraint, the framework learns scale-invariant features, enabling robust performance on real-world heavy rain. Experimental results demonstrate that FBL significantly outperforms existing state-of-the-art methods, particularly in scenarios with substantial scale mismatch.

Introduction

Most modern deraining methods are data-driven, relying on synthetic pairs of rainy and clean images. A major bottleneck is scale variance: models trained on small, needle-like streaks often fail when faced with the large, thick streaks common in real-world heavy downpours. Because synthesizing every possible rain scale is impractical, existing models lack generalization. This work aims to bridge this gap by creating a "scale-free" model that understands the intrinsic texture of rain streaks regardless of their physical size.

Methodology

The proposed framework introduces two primary innovations: a fractal-based network architecture and a self-supervised training strategy.

Fractal Band Learning (FBL)

Inspired by traditional signal processing concepts of frequency band recovery, the FBL network utilizes a recursive, fractal structure. It is composed of Refine Units (RU) that progressively improve feature maps and Fusion Units (FU) that integrate them. This hierarchical nesting allows the network to capture complex dependencies across different frequency bands more effectively than standard flat architectures.

Scale-Free Self-Supervision

To achieve scale invariance, the authors incorporate a self-supervised constraint during training. The network processes images at their original scale alongside various rescaled (zoomed) versions. The training loss enforces consistency between features extracted from these different scales, teaching the network to respond to the texture of rain rather than its absolute pixel dimensions.

The complete FBL framework, integrating Fractal Band Learning modules with a Scale-Free Self-Supervision pipeline for robust deraining.

Experimental Evaluation

The model was evaluated on several benchmarks, including Rain100L, Rain100H, and Rain800. To specifically test scale robustness, the authors introduced Rain100H-S2 and Rain100H-S3, where rain streaks are 2x and 3x larger than those in the training set.

Quantitative Results

FBL significantly outperformed state-of-the-art methods. On Rain100H, it achieved a 30.47 dB PSNR, a substantial improvement over previous leaders like JORDER (24.10 dB) and RESCAN (26.06 dB). Most notably, on the Rain100H-S3 scale-mismatch test, the self-supervised version (FBL-SS) achieved 20.85 dB, while standard models like DetailNet dropped to 15.03 dB.

Qualitative Performance

Qualitative tests on real-world images demonstrate that FBL-SS can remove large, obstructive streaks that other methods typically ignore or blur. Comparison with input images shows that the model preserves background details while effectively isolating and removing rain artifacts.

Qualitative result showing the effectiveness of the FBL-SS model in removing rain streaks from a real-world scene.

Conclusion

Fractal Band Learning provides a robust solution to the scale variance problem in image deraining. By combining recursive feature refinement with cross-scale self-supervision, the model achieves superior generalization to diverse real-world conditions. Future work will explore the use of adversarial losses to further preserve fine background details and reduce potential blurring in extreme cases.

Summary updated

Abstract

Data-driven rain streak removal methods, which most of rely on synthesized paired data, usually come across the generalization problem when being applied in real cases. In this paper, we propose a novel deep-learning based rain streak removal method injected with self-supervision to improve the ability to remove rain streaks in various scales. To realize this goal, we made efforts in two aspects. First, considering that rain streak removal is highly correlated with texture characteristics, we create a fractal band learning (FBL) network based on frequency band recovery. It integrates commonly seen band feature operations with neural modules and effectively improves the capacity to capture discriminative features for deraining. Second, to further improve the generalization ability of FBL for rain streaks in various scales, we add cross-scale self-supervision to regularize the network training. The constraint forces the extracted features of inputs in different scales to be equivalent after rescaling. Therefore, FBL can offer similar responses based on solely image content without the interleave of scale and is capable to remove rain streaks in various scales. Extensive experiments in quantitative and qualitative evaluations demonstrate the superiority of our FBL for rain streak removal, especially for the real cases where very large rain streaks exist, and prove the effectiveness of its each component. Our code will be public available at: https://github.com/flyywh/AAAI-2020-FBL-SS.
BibTeX
@inproceedings{yang2020towards,
  author = {Wenhan Yang and Shiqi Wang and Dejia Xu and Xiaodong Wang and Jiaying Liu},
  title = {{Towards Scale-Free Rain Streak Removal via Self-Supervised Fractal Band Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6954},
}
Figure