Multi-task Image Restoration Guided By Robust DINO Features

Xin LinarXiv 2023 · DOI · arXiv

Abstract

DINO-IR is a novel multi-task image restoration framework that leverages robust, degradation-insensitive features from the DINOv2 foundation model. By employing these semantic "anchors" to guide the restoration process, the model achieves state-of-the-art performance across denoising, deblurring, deraining, and dehazing. Crucially, DINO-IR maintains significantly higher stability than previous methods as the number of tasks increases, effectively mitigating the performance drop typically associated with multi-task interference.

Introduction

Image restoration is a fundamental computer vision challenge aimed at recovering high-quality images from versions corrupted by noise, blur, rain, or haze. While specialized models for individual tasks have achieved high performance, there is a growing shift toward unified multi-task models for improved efficiency. However, multi-tasking often introduces "task interference," where the optimization goals of different degradations conflict. For example, denoising seeks to smooth high-frequency noise, while deblurring aims to sharpen high-frequency details. As more tasks are added to a single model, performance typically degrades. DINO-IR addresses this by utilizing degradation-agnostic semantic representations as stable anchors to guide the restoration process.

Methodology

The DINO-IR framework integrates the robust feature extraction capabilities of DINOv2—a large-scale self-supervised vision transformer—into a restoration backbone, specifically Restormer. The architecture is designed around three core components:

  1. Pixel-Semantic Fusion (PSF) Module: This module exploits the hierarchical nature of DINOv2. Shallow layers provide fine-grained spatial details, while deeper layers offer high-level semantic context. A gating network dynamically weights these features, selecting the most relevant information for the specific restoration task at hand.
  2. DINO-Restore (D-R) Adaption and Fusion: This component bridges the gap between the foundation model and the restoration pipeline. It adapts DINO features to the required dimensions and utilizes a self-attention mechanism to fuse them effectively into the main restoration flow.
  3. DINO Perception Contrastive (DPC) Loss: During training, this loss function encourages the restored image's features to align with the clean ground truth while distancing them from the degraded input in the feature space.
The DINO-IR architecture integrates DINOv2 features into a restoration backbone via semantic fusion and adaptation modules.

Key Insights: DINOv2 as a Robust Anchor

The central innovation of DINO-IR is the use of DINOv2 features as a "robust anchor." Empirical analysis reveals that these high-level features remain remarkably consistent regardless of whether an image is clean, rainy, or blurry. This stability allows the model to maintain structural integrity during restoration, even under heavy corruption. Furthermore, by incorporating semantic guidance, the model moves beyond local pixel-level operations to understand underlying object shapes and textures, leading to more accurate and realistic reconstructions than models relying solely on local neighborhoods.

Experimental Results and Stability

The model was evaluated on a comprehensive suite of standard benchmarks, including DID and Rain100L for deraining, SOTS for dehazing, and Urban100 for denoising. DINO-IR consistently outperformed existing multi-task leaders such as AirNet and PromptIR, showing an average improvement of 0.13 dB to 1 dB in PSNR across various tasks. A particularly notable milestone was achieved in dehazing, where the model gained up to 3.41 dB over previous multi-task methods.

A key strength of DINO-IR is its "enhanced stability." While traditional multi-task models see a sharp decline in performance as the number of tasks grows, DINO-IR exhibits much greater resilience. This is visualized in the comparison between the base Restormer and the DINO-guided version, where the performance gap widens as more tasks are added.

DINO-IR shows superior performance and stability compared to Restormer as the number of restoration tasks increases.

Conclusion and Future Work

DINO-IR represents a significant advancement in robust, general-purpose image restoration. By grounding the restoration process in stable semantic features, it overcomes the traditional pitfalls of multi-task interference. While the model shows superior quality and generalization, the use of a large foundation model like DINOv2 introduces computational overhead. Future research will likely explore methods to reduce this cost and evaluate the framework's performance on complex, "in-the-wild" camera artifacts that go beyond synthetic degradations.

Summary updated

Abstract

Multi-task image restoration has gained significant interest due to its inherent versatility and efficiency compared to its single-task counterpart. However, performance decline is observed with an increase in the number of tasks, primarily attributed to the restoration model's challenge in handling different tasks with distinct natures at the same time. Thus, a perspective emerged aiming to explore the degradation-insensitive semantic commonalities among different degradation tasks. In this paper, we observe that the features of DINOv2 can effectively model semantic information and are independent of degradation factors. Motivated by this observation, we propose \mbox{\textbf{DINO-IR}}, a multi-task image restoration approach leveraging robust features extracted from DINOv2 to solve multi-task image restoration simultaneously. We first propose a pixel-semantic fusion (PSF) module to dynamically fuse DINOV2's shallow features containing pixel-level information and deep features containing degradation-independent semantic information. To guide the restoration model with the features of DINOv2, we develop a DINO-Restore adaption and fusion module to adjust the channel of fused features from PSF and then integrate them with the features from the restoration model. By formulating these modules into a unified deep model, we propose a DINO perception contrastive loss to constrain the model training. Extensive experimental results demonstrate that our DINO-IR performs favorably against existing multi-task image restoration approaches in various tasks by a large margin. The source codes and trained models will be made available.
BibTeX
@misc{lin2023multi,
  author = {Xin Lin},
  title = {{Multi-task Image Restoration Guided By Robust DINO Features}},
  year = {2023},
  eprint = {2312.01677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.01677},
}
Figure