TL;DR
This paper introduces Dual Prompt Learning (DPL), a novel continual learning approach for single image rain removal. DPL uses compact, learnable "prompts" at both image and feature levels to guide a transformer-based deraining model, effectively preventing catastrophic forgetting without requiring task identity or increasing model size significantly.
What Problem Does This Paper Address?
Single image deraining has seen significant progress with deep learning, particularly transformer-based models. However, these models often struggle when faced with diverse rain conditions that appear sequentially (continual learning). This is due to "catastrophic forgetting," where learning a new rain type causes the model to forget how to handle previous types. Existing continual learning methods are often computationally expensive, require storing past data, or lead to a linear increase in model parameters. This paper aims to address this by developing an efficient continual learning method for rain removal that maintains performance across different rain types without these drawbacks.
Methods
The core of DPL lies in its "dual prompt learning" mechanism. Instead of directly fine-tuning the entire model for each new rain type, DPL introduces small, learnable parameter sets called "prompts." These prompts act as task-specific instructions for a frozen transformer backbone.
Here's a breakdown of the key components:
- Dual Prompt Pools: DPL maintains two pools of prompts: one for the image level and one for the feature level. Each prompt is a key-value pair, where the key is used for matching and the value is a learnable representation.
- Instance-wise Prompt Selection: For each input image, DPL dynamically selects a small subset of top-N prompts from both pools. This selection is based on a similarity measure (cosine distance) between the input (or its feature representation) and the prompt keys. This allows the model to adapt to specific rain characteristics without needing to know the explicit task identity.
- Concatenation for Guidance: The selected image-level prompts are concatenated with the input image embeddings, and the selected feature-level prompts are concatenated with the transformer's feature representations. This injects task-specific knowledge directly into the model's processing pipeline.
- Joint Regularization: To further prevent forgetting, DPL incorporates a regularization term that penalizes significant changes to both the model's original parameters and the learned prompts. This ensures that knowledge from previous tasks is preserved.
This approach is designed to be parameter-efficient, as only the prompts and a small regularization component are updated, while the main transformer backbone remains largely fixed.
Datasets
The paper evaluates DPL on three widely-used single image rain removal datasets:
- Rain800: Contains 600 training and 200 testing images.
- Rain100H: Contains 1,800 training and 100 testing images.
- Rain100L: Contains 1,800 training and 100 testing images.
The experiments involve sequential training on task sequences like Rain800-Rain100H and Rain800-Rain100L to simulate the continual learning scenario.
Key Insights and Ideas
- Prompts as Task Memory: The paper demonstrates that compact, learnable prompts can effectively store and retrieve task-specific knowledge for continual learning, acting as a more efficient alternative to full model fine-tuning or parameter isolation.
- Image and Feature Level Synergy: Applying prompts at both the input (image) and intermediate (feature) levels proves crucial for leveraging transferred knowledge across different stages of the deraining process.
- Task-Agnostic Adaptation: DPL's ability to select prompts based on input similarity, rather than explicit task labels, makes it highly adaptable to real-world scenarios where task boundaries might be unknown or ambiguous.
- Competitive Performance on Static Data: Surprisingly, DPL also achieves superior results on stationary datasets compared to other methods, suggesting that its prompt-based approach inherently improves deraining quality even without the continual learning constraint.
Key Results
- DPL consistently outperforms existing continual learning methods on sequential rain removal tasks, achieving higher PSNR and SSIM scores.
- In the Rain800-Rain100H sequence, DPL achieves a PSNR of 24.39 dB and SSIM of 0.8365, significantly outperforming methods like EWC, Replay, and PIGWM.
- Even when applied to a single dataset (Rain800), DPL achieves a PSNR of 27.52 dB and SSIM of 0.8667, surpassing the baseline.
- The proposed method adds only a small percentage (5.2%) of extra parameters compared to the baseline, highlighting its parameter efficiency.
- Ablation studies confirm that both image and feature prompts are essential, with feature-level prompts playing a slightly more critical role. Parameter regularization further boosts performance.
Limitations and Future Work
- Computational Cost: While more efficient than some methods, the prompt selection process still adds computational overhead.
- Hyperparameter Tuning: The performance of DPL is influenced by hyperparameters like the number of prompts (N) and the weighting of different loss terms.
- Future Work: The authors suggest exploring more sophisticated prompt selection mechanisms and investigating the application of DPL to other low-level vision tasks that involve continual learning. They also mention the potential for further optimization of the prompt design and regularization strategies.
Summary updated