UniRain: Unified Image Deraining with RAG-based Dataset Distillation and Multi-objective Reweighted Optimization

Qianfeng Yang, Qiyuan Guan, Xiang Chen, Jiyu Jin, Guiyue Jin, Jiangxin Dong

TL;DR

UniRain is a unified image deraining framework that simultaneously handles rain streaks and raindrops across both daytime and nighttime conditions. By combining a novel Retrieval-Augmented Generation (RAG) dataset distillation pipeline with a multi-objective reweighted optimization strategy, it overcomes data quality disparities and task imbalances to deliver state-of-the-art restoration performance with high computational efficiency.

What Problem Does This Paper Address?

Most deep learning-based image deraining methods are hyper-specialized. A model trained to remove daytime rain streaks will often fail miserably when faced with nighttime raindrops, where low illumination and complex light reflections alter the degradation pattern.

To build an "all-in-one" universal derainer, the intuitive approach is to simply merge all public datasets. However, the authors identify two major roadblocks to this strategy:

  1. Data Quality Disparity: Public datasets vary wildly in realism and background quality. Directly mixing them introduces noisy, inaccurate supervisory signals that hinder model convergence.
  2. Task Imbalance: Different rain types present different learning difficulties. During joint training, a standard loss function causes the model to overfit to easier tasks (like nighttime streaks) while neglecting more complex ones (like daytime raindrops).
Overview of motivation showing data quality disparities and training imbalances across different rain types.

Methods

The authors introduce UniRain, a framework consisting of three core innovations designed to solve the dataset and optimization challenges of unified deraining:

1. RAG-based Dataset Distillation

Instead of using raw, noisy datasets, the authors design an offline dataset distillation pipeline. For every training image, the system uses a hierarchical similarity matching process (textual captions via BLIP, visual features via CLIP, and structural similarity via SSIM) to retrieve the most relevant real-world rainy images from a curated database. An ensemble of Vision-Language Models (VLMs)—including InternVL2.5, LLaVA-NeXT, and MobileVLM—then evaluates the query image against these real-world references to filter out low-fidelity or unrealistic training samples.

2. Asymmetric Mixture-of-Experts (MoE) Architecture

To handle diverse rain types efficiently, UniRain adopts an asymmetric MoE structure:

  • Soft-MoE Encoder: Uses continuous routing weights to softly combine expert representations, ensuring that diverse degradation cues are collaboratively preserved.
  • Hard-MoE Decoder: Employs discrete Top-$k$ routing to activate only the most relevant experts, focusing computational power on sharp, fine-grained texture reconstruction.

3. Multi-objective Reweighted Optimization

To prevent the model from favoring easier rain types, the authors introduce a dynamic loss-reweighting mechanism. By tracking the Type Balance Score (TBS) (how fast a task is converging) and the Type Stability Score (TSS) (how consistently it converges) within a sliding temporal window, the optimizer dynamically boosts the loss weights of lagging or unstable tasks, ensuring synchronized learning.

The overall framework of UniRain, detailing the RAG-based dataset distillation pipeline and the asymmetric MoE architecture with multi-objective optimization.

Datasets

  • RainRAG (New): The authors curated this high-quality dataset by running their RAG distillation pipeline over more than 2 million public synthetic and real-world image pairs. The pipeline retained only the top ~2.6% highest-quality samples, resulting in 52,869 training pairs split into four categories: daytime rain streaks (DRS), daytime raindrops (DRD), nighttime rain streaks (NRS), and nighttime raindrops (NRD).
  • Evaluation Benchmarks: The model was tested on the RainRAG test set, as well as three real-world public benchmarks: RealRain-1k, RainDS-real, and WeatherBench.

Key Insights and Ideas

  • Quality Over Quantity: A key takeaway from the ablation studies is that training on a massive, unfiltered dataset (~2,000k images) yields worse results than training on a clean, distilled subset (~52k images). In low-level vision, data realism and consistency matter more than sheer volume.
  • Stability Matters as Much as Speed: Traditional multi-task optimization often balances tasks based solely on their instantaneous loss or convergence speed. UniRain demonstrates that incorporating a historical stability metric prevents the optimizer from over-allocating weights to tasks that are wildly oscillating or diverging.
  • Asymmetry is Efficient: Combining soft routing in the encoder with hard routing in the decoder strikes an ideal balance between broad feature exploration and sharp, localized feature exploitation.

Key Results

  • Quantitative Performance: On the RainRAG dataset, UniRain achieved an average PSNR of 28.93 dB, outperforming the previous state-of-the-art universal method, URIR, by 1.02 dB. On real-world benchmarks, it achieved a substantial average improvement of 1.73 dB PSNR over URIR.
  • Computational Efficiency: Despite its superior performance, UniRain is highly efficient, requiring only 126.54 GFLOPs and 24.38M parameters, making it significantly lighter than specialized architectures like DRSformer (220.38 GFLOPs, 33.62M parameters).
  • Downstream Generalization: The model showed robust zero-shot generalization across diverse unseen operational domains, including autonomous driving, unmanned aerial vehicles (UAVs), and maritime environments.
Visual comparison of deraining results, demonstrating UniRain's ability to preserve structural details and remove complex artifacts compared to other SOTA methods.

Limitations and Future Work

While the authors successfully extended UniRain to an "all-in-one" weather restoration task (handling rain, snow, and haze simultaneously), the framework treats different weather conditions as independent optimization objectives. In the real world, weather conditions often overlap (e.g., rainy and foggy conditions occurring together). Future work could explore modeling the physical and feature-level interactions between concurrent weather degradations rather than treating them as isolated tasks.

Summary updated

Abstract

Despite significant progress has been made in image deraining, we note that most existing methods are often developed for only specific types of rain degradation and fail to generalize across diverse real-world rainy scenes. How to effectively model different rain degradations within a universal framework is important for real-world image deraining. In this paper, we propose UniRain, an effective unified image deraining framework capable of restoring images degraded by rain streak and raindrop under both daytime and nighttime conditions. To better enhance unified model generalization, we construct an intelligent retrieval augmented generation (RAG)-based dataset distillation pipeline that selects high-quality training samples from all public deraining datasets for better mixed training. Furthermore, we incorporate a simple yet effective multi-objective reweighted optimization strategy into the asymmetric mixture-of-experts (MoE) architecture to facilitate consistent performance and improve robustness across diverse scenes. Extensive experiments show that our framework performs favorably against the state-of-the-art models on our proposed benchmarks and multiple public datasets.
BibTeX
@inproceedings{yang2026unirain,
  author = {Qianfeng Yang and Qiyuan Guan and Xiang Chen and Jiyu Jin and Guiyue Jin and Jiangxin Dong},
  title = {{UniRain: Unified Image Deraining with RAG-based Dataset Distillation and Multi-objective Reweighted Optimization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=PKlYYKNUZv},
}
Figure