Abstract
Most existing image de-raining networks learn fixed mapping rules between paired rainy/clean images on a single synthetic dataset, remaining static thereafter. However, because a single synthetic dataset provides only a partial view of rain streak distributions, models often overfit to these biased distributions and fail to generalize to complex real-world rainy scenes. In this paper, we propose a method to accumulate de-raining knowledge from multiple synthetic datasets onto a single network parameter set. We introduce Neural Reorganization (NR), which maintains a stability-plasticity trade-off by borrowing synaptic consolidation mechanisms from the biological brain and utilizing knowledge distillation. Our NR algorithm allows a deep model to be trained on a sequence of synthetic rainy datasets while overcoming catastrophic forgetting. Extensive experiments demonstrate that our method achieves state-of-the-art results on real-world rainy images by successfully accumulating diverse de-raining knowledge.
Introduction
Single-image de-raining is a fundamental task in computer vision, yet deep learning models in this domain face a significant "generalization gap." While models perform exceptionally well on the synthetic datasets they were trained on, they often fail when applied to real-world rain, which varies significantly in density, orientation, and appearance.
A major hurdle in improving generalization is catastrophic forgetting. When a model is trained on a new dataset (e.g., transitioning from light rain to heavy rain), the new knowledge often overwrites the old, causing performance on previous tasks to degrade. Simple strategies like mixing all datasets together ("Mix" strategy) often lead to biased results or gradient interference. To address this, we propose Neural Reorganization (NR), a framework that enables a single network to learn from a sequence of datasets, accumulating knowledge to become a "General-version Image De-rainer" (GID).
Neural Reorganization Framework
The NR framework is designed to manage the stability-plasticity trade-off—the ability to acquire new information (plasticity) while preserving existing knowledge (stability). It consists of two primary components:
Synaptic Consolidation (SC)
Inspired by the biological brain, where important synaptic connections are consolidated and shielded from change, NR calculates an "importance weight" for every network parameter based on gradients. We employ a Dual-Exponent module, which mathematically models how brain plasticity decreases over time, to create a boolean mask. This mask protects the most critical parameters for previously learned tasks, preventing them from being updated during training on new datasets.
Knowledge Distillation (KD)
To further mitigate forgetting, we incorporate a distillation term into the loss function. When training on a new dataset, the current model is encouraged to produce outputs similar to its previous iteration. This ensures that the underlying logic and feature representations learned from earlier datasets are preserved, even as the model adapts to new rain distributions.
Experimental Setup
To evaluate the generalization power of NR, we trained the model exclusively on synthetic data and tested it on real-world datasets:
- Synthetic Training Sets: Rain100H, Rain100L, Rain800, and Rain14000.
- Real-World Evaluation:
- SPA-Data: 1,000 real-world rainy/clean pairs for quantitative assessment.
- Real-Internet: 146 web-collected images for qualitative testing.
Results and Analysis
Quantitative Performance
On the real-world SPA-Data, the NR-equipped model achieved a PSNR of 35.60 dB. This significantly outperforms the "Mix" strategy (34.85 dB) and the sequential training baseline without protection (34.91 dB). These results indicate that sequential learning with proper consolidation is more effective than simple data mixing, likely due to reduced gradient interference.
Qualitative Generalization
Visual comparisons highlight the robustness of the NR approach. While standard models trained on single datasets often leave heavy streaks or introduce blurring artifacts in real-world scenes, the NR model effectively removes diverse rain types while maintaining sharp background details.
As shown in the figure above, models trained on specific synthetic datasets (e.g., Rain100H or Rain100L) struggle when the real-world rain distribution deviates from their training data. In contrast, our Neural Reorganization method successfully generalizes across different cases, producing cleaner and more visually pleasing results.
Conclusion
Neural Reorganization offers a novel perspective on de-raining by focusing on knowledge accumulation rather than just architectural innovation. By mimicking the brain's synaptic consolidation, we enable models to learn from a sequence of synthetic datasets without forgetting. This approach creates a more versatile de-rainer capable of handling the complexities of real-world weather. Future work will explore the sensitivity of the training sequence and potential dynamic architecture expansions to further increase network capacity.
Summary updated