Abstract
RAM++ is a novel "all-in-one" image restoration framework designed to address multiple types of degradation—including noise, rain, haze, and blur—within a single model. Unlike traditional degradation-oriented approaches that attempt to classify and subtract specific noise patterns, RAM++ shifts the focus toward learning the intrinsic content of natural images. By employing adaptive masked pre-training and DINOv2-guided fine-tuning, the framework achieves state-of-the-art, balanced performance across both seen and unseen degradations, demonstrating superior robustness in extreme scenarios.
Introduction
In real-world applications, images are frequently corrupted by environmental or hardware factors such as rain, low light, motion blur, or sensor noise. While specialized models exist for individual tasks, "all-in-one" models aim to handle multiple degradations simultaneously. However, current all-in-one models are primarily degradation-oriented, attempting to map specific corruptions into distinct feature spaces. This often leads to three critical issues: insufficient generative capability in extreme scenarios, unbalanced performance across different tasks (where improving one task degrades another), and weak generalization to unseen or mixed degradations.
RAM++ addresses these flaws by proposing that models should learn what a clean image looks like rather than what degradation looks like. This shift from degradation-oriented to content-oriented learning allows for more robust and generalized image restoration.
Methods
RAM++ utilizes a two-stage framework consisting of pre-training and fine-tuning, centered on three core innovations:
- Adaptive Semantic-Aware Mask (AdaSAM) Pre-training: Standard Masked Image Modeling (MIM) often uses large patches that destroy local textures essential for restoration. AdaSAM introduces adaptive, pixel-level masking focused on semantically rich and highly textured regions. This forces the model to reconstruct complex structures, building strong generative image priors.
- Mask Attribute Conductance (MAC): To bridge the distribution gap between masked images (pre-training) and full images (inference), MAC uses a gradient-based analysis to identify critical network layers. By selectively fine-tuning only the top ~30% of these layers, the model adapts to full images while preserving the powerful priors learned during pre-training.
- Robust Feature Regularization (RFR): During fine-tuning, RAM++ integrates representations from DINOv2, a pre-trained vision foundation model. DINOv2 features are highly semantic and naturally invariant to low-level degradations, helping RAM++ maintain structural coherence even under extreme corruption.
Key Results and Evaluation
The authors evaluated RAM++ across multiple benchmarks, including 3-task and 7-task settings, as well as Out-of-Distribution (OOD) and extreme degradation scenarios (e.g., CDD-11, UIEB).
- State-of-the-Art Performance: In a comprehensive 7-task setting, RAM++ improved average PSNR by 0.70 dB over existing best methods.
- Balanced Robustness: As shown in the radar chart, RAM++ maintains high performance across diverse conditions without the typical trade-offs seen in previous models. It achieved a massive 1.84 dB PSNR gain over the Restormer baseline when tested on completely unseen noise types (e.g., speckle noise).
- Efficiency: Using MAC to fine-tune just 30% of the network, RAM++ still outperformed fully trained competitive models, highlighting the effectiveness of its pre-trained representation.
Conclusion
RAM++ demonstrates that prioritizing intrinsic image content over specific degradation patterns leads to more stable and generalized image restoration. By combining adaptive masking with foundation model guidance, it sets a new benchmark for all-in-one restoration. Future work will focus on optimizing data-mixing strategies and extending the framework to video restoration to ensure temporal consistency across frames.
Summary updated