Improving De-raining Generalization via Neural Reorganization

Jie Xiao, Man Zhou, Xueyang Fu, Aiping Liu, Zheng-Jun ZhaICCV 2021 · OpenReview

Abstract

Most existing image de-raining networks learn fixed mapping rules between paired rainy/clean images on a single synthetic dataset, remaining static thereafter. However, because a single synthetic dataset provides only a partial view of rain streak distributions, models often overfit to these biased distributions and fail to generalize to complex real-world rainy scenes. In this paper, we propose a method to accumulate de-raining knowledge from multiple synthetic datasets onto a single network parameter set. We introduce Neural Reorganization (NR), which maintains a stability-plasticity trade-off by borrowing synaptic consolidation mechanisms from the biological brain and utilizing knowledge distillation. Our NR algorithm allows a deep model to be trained on a sequence of synthetic rainy datasets while overcoming catastrophic forgetting. Extensive experiments demonstrate that our method achieves state-of-the-art results on real-world rainy images by successfully accumulating diverse de-raining knowledge.

Introduction

Single-image de-raining is a fundamental task in computer vision, yet deep learning models in this domain face a significant "generalization gap." While models perform exceptionally well on the synthetic datasets they were trained on, they often fail when applied to real-world rain, which varies significantly in density, orientation, and appearance.

A major hurdle in improving generalization is catastrophic forgetting. When a model is trained on a new dataset (e.g., transitioning from light rain to heavy rain), the new knowledge often overwrites the old, causing performance on previous tasks to degrade. Simple strategies like mixing all datasets together ("Mix" strategy) often lead to biased results or gradient interference. To address this, we propose Neural Reorganization (NR), a framework that enables a single network to learn from a sequence of datasets, accumulating knowledge to become a "General-version Image De-rainer" (GID).

Neural Reorganization Framework

The NR framework is designed to manage the stability-plasticity trade-off—the ability to acquire new information (plasticity) while preserving existing knowledge (stability). It consists of two primary components:

Synaptic Consolidation (SC)

Inspired by the biological brain, where important synaptic connections are consolidated and shielded from change, NR calculates an "importance weight" for every network parameter based on gradients. We employ a Dual-Exponent module, which mathematically models how brain plasticity decreases over time, to create a boolean mask. This mask protects the most critical parameters for previously learned tasks, preventing them from being updated during training on new datasets.

Knowledge Distillation (KD)

To further mitigate forgetting, we incorporate a distillation term into the loss function. When training on a new dataset, the current model is encouraged to produce outputs similar to its previous iteration. This ensures that the underlying logic and feature representations learned from earlier datasets are preserved, even as the model adapts to new rain distributions.

Experimental Setup

To evaluate the generalization power of NR, we trained the model exclusively on synthetic data and tested it on real-world datasets:

  • Synthetic Training Sets: Rain100H, Rain100L, Rain800, and Rain14000.
  • Real-World Evaluation:
    • SPA-Data: 1,000 real-world rainy/clean pairs for quantitative assessment.
  • Real-Internet: 146 web-collected images for qualitative testing.

Results and Analysis

Quantitative Performance

On the real-world SPA-Data, the NR-equipped model achieved a PSNR of 35.60 dB. This significantly outperforms the "Mix" strategy (34.85 dB) and the sequential training baseline without protection (34.91 dB). These results indicate that sequential learning with proper consolidation is more effective than simple data mixing, likely due to reduced gradient interference.

Qualitative Generalization

Visual comparisons highlight the robustness of the NR approach. While standard models trained on single datasets often leave heavy streaks or introduce blurring artifacts in real-world scenes, the NR model effectively removes diverse rain types while maintaining sharp background details.

Qualitative comparison of de-raining performance on real-world scenes.

As shown in the figure above, models trained on specific synthetic datasets (e.g., Rain100H or Rain100L) struggle when the real-world rain distribution deviates from their training data. In contrast, our Neural Reorganization method successfully generalizes across different cases, producing cleaner and more visually pleasing results.

Conclusion

Neural Reorganization offers a novel perspective on de-raining by focusing on knowledge accumulation rather than just architectural innovation. By mimicking the brain's synaptic consolidation, we enable models to learn from a sequence of synthetic datasets without forgetting. This approach creates a more versatile de-rainer capable of handling the complexities of real-world weather. Future work will explore the sensitivity of the training sequence and potential dynamic architecture expansions to further increase network capacity.

Summary updated

Abstract

Most existing image de-raining networks could only learn fixed mapping rules between paired rainy/clean images on single synthetic dataset and then stay static for lifetime. However, since single synthetic dataset merely provides a partial view for the distribution of rain streaks, deep models well trained on an individual synthetic dataset tend to overfit on this biased distribution. This leads to the inability of these methods to well generalize to complex and changeable real-world rainy scenes, thus limiting their practical applications. In this paper, we try for the first time to accumulate the de-raining knowledge from multiple synthetic datasets on a single network parameter set to improve the de-raining generalization of deep networks. To achieve this goal, we explore Neural Reorganization (NR) to allow the de-raining network to keep a subtle stability-plasticity trade-off rather than naive stabilization after training phase. Specifically, we design our NR algorithm by borrowing the synaptic consolidation mechanism in the biological brain and knowledge distillation. Equipped with our NR algorithm, the deep model can be trained on a list of synthetic rainy datasets by overcoming catastrophic forgetting, making it a general-version de-raining network. Extensive experimental validation shows that due to the successful accumulation of de-raining knowledge, our proposed method can not only process multiple synthetic datasets consistently, but also achieve state-of-the-art results when dealing with real-world rainy images.
BibTeX
@inproceedings{xiao2021improving,
  author = {Jie Xiao and Man Zhou and Xueyang Fu and Aiping Liu and Zheng-Jun Zha},
  title = {{Improving De-raining Generalization via Neural Reorganization}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=0Wj8UjI0B0},
}
Figure