MatRes: Zero-Shot Test-Time Model Adaptation for Simultaneous Matching and Restoration

Kanggeon Lee, Soochahn Lee, Kyoung Mu LeearXiv 2026 · arXiv

TL;DR

MatRes is a zero-shot test-time adaptation framework that simultaneously tackles image restoration and geometric matching for image pairs with wide viewpoints and varying quality. By using a lightweight adapter to enable mutual guidance between frozen, pretrained networks, it significantly improves both alignment and image quality at inference time without requiring any offline training data.

Figure 1: Zero-shot Test-Time Adaptation. MATRES enhances degraded and viewpoint-shifted image pairs by leveraging the mutual guidance of a matching and a restoration network to produce a restored and aligned output image.

What Problem Does This Paper Address?

When capturing multiple photos of a scene from different angles, it is common for some images to suffer from degradations like blur, noise, or low resolution, while others remain sharp. Intuitively, we should be able to use the high-quality (HQ) image to help restore the low-quality (LQ) one.

However, doing this requires solving two coupled problems: geometric matching (aligning the two different viewpoints) and image restoration. When treated independently, these tasks interfere with each other: severe image degradation breaks matching algorithms, and spatial misalignment prevents restoration networks from using the reference image properly. While prior works address joint optimization for video or burst photos (which have tiny viewpoint changes), doing this for wide-baseline image pairs is incredibly difficult because collecting ground-truth training data (clean images, exact transformation fields, and known degradation kernels) is practically impossible.

Methods

MatRes introduces a zero-shot test-time adaptation mechanism that optimizes parameters directly during inference on a single pair of images.

Figure 2: Method Overview. Given an input pair (ILQ, IHQ), the pretrained generative prior model Mθ and the pretrained restoration network Rϕ remain frozen during adaptation (gra- dient flows through both models but their parameters are not updated). The zero-initialized adapter Hψ is the only train- able module, refined iteratively using losses computed on the IEQ and IHQ image pairs. At each adaptation step, Mθ extracts generative priors and estimates the transformation TLQ→HQ, the ILQ image are warped and injected into Rϕ, and the adapter Hψ learns to align and modulate features for improved restora- tion.

The architecture consists of three components:

  1. Pretrained Matching Network ($\mathcal{M}_\theta$): Kept frozen. It extracts generative feature priors from the LQ and HQ images and estimates the geometric transformation field to warp the LQ image.
  2. Pretrained Restoration Network ($\mathcal{R}_\phi$): Kept frozen. It takes the warped LQ image and the HQ reference image to produce an enhanced image ($\mathcal{I}_{EQ}$).
  3. Lightweight Feature Adapter ($\mathcal{H}_\psi$): The only trainable module. It is a simple convolutional network initialized to zero, meaning it initially has no effect on the restoration network.

During test-time optimization, the matching network's features are passed through the adapter and injected into the restoration network via additive modulation. The adapter parameters ($\psi$) are iteratively updated using a two-part loss function calculated only within the overlapping warped regions:

  • Off-Diagonal Loss: Penalizes high off-diagonal responses in the feature cost volume, forcing the network to find sharp, distinct point correspondences.
  • Pixel-Matching Loss: Minimizes the mean squared error (MSE) between the restored image and the HQ reference image.

Datasets

The authors evaluated MatRes across three distinct restoration tasks using synthetic viewpoint shifts and illumination variations applied to standard benchmarks:

  • Super-Resolution (SR): RealSR, DIV2K, Set5, Set14, Urban100, and BSD100.
  • Denoising: Set5, Set14, and Urban100.
  • Deblurring: GoPro, HIDE, RealBlur-J, RealBlur-R, and CelebA.

No offline training data was used, as the framework adapts entirely at test time.

Key Insights and Ideas

  • The Virtuous Cycle of Mutual Guidance: The core "aha moment" is that matching and restoration can form a cooperative loop. Better geometric alignment allows the restoration network to extract high-fidelity details from the reference image. Conversely, as the restoration network cleans up the image, it provides sharper features back to the matching network, drastically improving alignment accuracy.
  • Zero-Initialization for Stability: Initializing the adapter weights to zero ensures that the framework starts from the baseline performance of the strong, pretrained models. Test-time optimization then safely injects informative corrections without destroying the models' existing capabilities.
  • Repurposing Foundation Models: Instead of training massive joint networks from scratch, MatRes proves that the emergent generative features inside large-scale matching models (like diffusion or vision transformer backbones) can be successfully projected into completely different restoration pipelines using a tiny fraction of learnable parameters.

Key Results

MatRes achieved substantial quantitative and qualitative gains across all tasks compared to using standalone restoration or matching models.

Figure 3: Qualitative Restoration Results. Restoration performance is compared with and without the matching network Mθ across three tasks. For super-resolution (SR), we evaluate EDSR [33], ESRGAN [45], SwinIR [32], and HAT [46]; for denois- ing, we evaluate DnCNN [47], N2V [35], N2N [34], and Restormer [31]; and for deblurring, we evaluate DeblurGAN [36], SRN [37], Restormer [31], and NAFNet [48]. The green box indicates the zoomed-in region.
  • Super-Resolution Boost: When pairing the RoMa matcher with an EDSR restorer, MatRes boosted the matching score (mAUC) by +9.61, improved the peak signal-to-noise ratio (PSNR) by +4.80 dB, and increased the structural similarity index (SSIM) by +0.216.
  • Outperforming Task-Specific Baselines: MatRes consistently outperformed state-of-the-art task-specific networks (such as AutoLUT for SR, MAXIM for denoising, and Stripformer for deblurring). Because traditional restoration models only look at the single degraded input, they cannot compete with MatRes's ability to leverage a misaligned multi-view reference image.

Limitations and Future Work

The paper does not explicitly dedicate a section to limitations, but a clear trade-off of test-time optimization is computational overhead. Because the framework relies on iterative gradient descent steps for every single image pair at inference time, it is inherently slower than standard feed-forward networks. Future work could explore accelerating this optimization process or extending the framework to handle multi-image clusters rather than just pairs.

Summary updated

Abstract

Real-world image pairs often exhibit both severe degradations and large viewpoint changes, making image restoration and geometric matching mutually interfering tasks when treated independently. In this work, we propose MatRes, a zero-shot test-time adaptation framework that jointly improves restoration quality and correspondence estimation using only a single low-quality and high-quality image pair. By enforcing conditional similarity at corresponding locations, MatRes updates only lightweight modules while keeping all pretrained components frozen, requiring no offline training or additional supervision. Extensive experiments across diverse combinations show that MatRes yields significant gains in both restoration and geometric alignment compared to using either restoration or matching models alone. MatRes offers a practical and widely applicable solution for real-world scenarios where users commonly capture multiple images of a scene with varying viewpoints and quality, effectively addressing the often-overlooked mutual interference between matching and restoration.
BibTeX
@misc{lee2026matres,
  author = {Kanggeon Lee and Soochahn Lee and Kyoung Mu Lee},
  title = {{MatRes: Zero-Shot Test-Time Model Adaptation for Simultaneous Matching and Restoration}},
  year = {2026},
  eprint = {2604.10081},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.10081},
}

Loading related content...

Figure