TL;DR
MatRes is a zero-shot test-time adaptation framework that simultaneously tackles image restoration and geometric matching for image pairs with wide viewpoints and varying quality. By using a lightweight adapter to enable mutual guidance between frozen, pretrained networks, it significantly improves both alignment and image quality at inference time without requiring any offline training data.
What Problem Does This Paper Address?
When capturing multiple photos of a scene from different angles, it is common for some images to suffer from degradations like blur, noise, or low resolution, while others remain sharp. Intuitively, we should be able to use the high-quality (HQ) image to help restore the low-quality (LQ) one.
However, doing this requires solving two coupled problems: geometric matching (aligning the two different viewpoints) and image restoration. When treated independently, these tasks interfere with each other: severe image degradation breaks matching algorithms, and spatial misalignment prevents restoration networks from using the reference image properly. While prior works address joint optimization for video or burst photos (which have tiny viewpoint changes), doing this for wide-baseline image pairs is incredibly difficult because collecting ground-truth training data (clean images, exact transformation fields, and known degradation kernels) is practically impossible.
Methods
MatRes introduces a zero-shot test-time adaptation mechanism that optimizes parameters directly during inference on a single pair of images.
The architecture consists of three components:
- Pretrained Matching Network ($\mathcal{M}_\theta$): Kept frozen. It extracts generative feature priors from the LQ and HQ images and estimates the geometric transformation field to warp the LQ image.
- Pretrained Restoration Network ($\mathcal{R}_\phi$): Kept frozen. It takes the warped LQ image and the HQ reference image to produce an enhanced image ($\mathcal{I}_{EQ}$).
- Lightweight Feature Adapter ($\mathcal{H}_\psi$): The only trainable module. It is a simple convolutional network initialized to zero, meaning it initially has no effect on the restoration network.
During test-time optimization, the matching network's features are passed through the adapter and injected into the restoration network via additive modulation. The adapter parameters ($\psi$) are iteratively updated using a two-part loss function calculated only within the overlapping warped regions:
- Off-Diagonal Loss: Penalizes high off-diagonal responses in the feature cost volume, forcing the network to find sharp, distinct point correspondences.
- Pixel-Matching Loss: Minimizes the mean squared error (MSE) between the restored image and the HQ reference image.
Datasets
The authors evaluated MatRes across three distinct restoration tasks using synthetic viewpoint shifts and illumination variations applied to standard benchmarks:
- Super-Resolution (SR): RealSR, DIV2K, Set5, Set14, Urban100, and BSD100.
- Denoising: Set5, Set14, and Urban100.
- Deblurring: GoPro, HIDE, RealBlur-J, RealBlur-R, and CelebA.
No offline training data was used, as the framework adapts entirely at test time.
Key Insights and Ideas
- The Virtuous Cycle of Mutual Guidance: The core "aha moment" is that matching and restoration can form a cooperative loop. Better geometric alignment allows the restoration network to extract high-fidelity details from the reference image. Conversely, as the restoration network cleans up the image, it provides sharper features back to the matching network, drastically improving alignment accuracy.
- Zero-Initialization for Stability: Initializing the adapter weights to zero ensures that the framework starts from the baseline performance of the strong, pretrained models. Test-time optimization then safely injects informative corrections without destroying the models' existing capabilities.
- Repurposing Foundation Models: Instead of training massive joint networks from scratch, MatRes proves that the emergent generative features inside large-scale matching models (like diffusion or vision transformer backbones) can be successfully projected into completely different restoration pipelines using a tiny fraction of learnable parameters.
Key Results
MatRes achieved substantial quantitative and qualitative gains across all tasks compared to using standalone restoration or matching models.
- Super-Resolution Boost: When pairing the RoMa matcher with an EDSR restorer, MatRes boosted the matching score (mAUC) by +9.61, improved the peak signal-to-noise ratio (PSNR) by +4.80 dB, and increased the structural similarity index (SSIM) by +0.216.
- Outperforming Task-Specific Baselines: MatRes consistently outperformed state-of-the-art task-specific networks (such as AutoLUT for SR, MAXIM for denoising, and Stripformer for deblurring). Because traditional restoration models only look at the single degraded input, they cannot compete with MatRes's ability to leverage a misaligned multi-view reference image.
Limitations and Future Work
The paper does not explicitly dedicate a section to limitations, but a clear trade-off of test-time optimization is computational overhead. Because the framework relies on iterative gradient descent steps for every single image pair at inference time, it is inherently slower than standard feed-forward networks. Future work could explore accelerating this optimization process or extending the framework to handle multi-image clusters rather than just pairs.
Summary updated