TL;DR
UGD-FM is a new framework that denoises RGB images using unaligned near-infrared (NIR) guidance by framing the task as a conditional flow matching problem. By iteratively cleaning the image and refining spatial alignment, the model achieves state-of-the-art restoration in just one or two inference steps, even under severe noise and large-range camera misalignment.
What Problem Does This Paper Address?
In low-light photography, RGB images are often plagued by heavy noise. Near-infrared (NIR) sensors can capture cleaner structural details in the dark, making them excellent "guides" for denoising the RGB counterpart. However, most existing guided denoising methods assume the two images are perfectly aligned.
In the real world, RGB and NIR sensors are usually separate cameras on a mobile device, leading to spatial misalignment (parallax). Finding the right pixels to match between a clean NIR image and a heavily corrupted, noisy RGB image is a "chicken-and-egg" problem: you need a clean image to find accurate correspondences, but you need those correspondences to clean the image. Previous methods either fail under large misalignments or require expensive, dense searches that aren't practical for mobile hardware.
Methods
The authors propose UGD-FM (Unaligned image Guided Denoising via cross-modal conditional Flow Matching), which introduces three main innovations:
-
Conditional Flow Matching (CFM) Trajectory: Instead of a single-pass denoising, the authors formulate the process as a progressive transport from a noisy state to a clean state. As the image moves along this trajectory and becomes cleaner, the model can more accurately align the NIR guidance, which in turn provides better cues for further denoising.
Figure 1: Illustration of the cross-modal conditional flow matching trajectory of UGD-FM. Given a noisy RGB image and an unaligned NIR guidance image, UGD-FM progressively restores the RGB state toward the clean target, forming a mutual reinforcing process between image restoration and guidance aggregation. For clearer visualization, the brightness of the zoom-in patches is enhanced. -
Stage-Focused & Inference-Consistent Training: Standard flow matching can require many steps. To make it efficient, the authors identify two critical stages: a "highly corrupted" stage for coarse structure and a "relatively cleaner" stage for fine details. They focus training on these specific anchors, allowing the model to perform exceptionally well with only 1 or 2 inference steps.
-
Adaptive Sparse Guidance Propagation: To handle large-range 2D misalignment without the massive computational cost of a dense search, the model uses a coarse-to-fine strategy. It identifies a small set of high-confidence "Top-K" candidates at a low resolution and propagates only those candidates to finer scales for refinement.
Figure 2: Network architecture of UGD-FM. (a) UGD-FM adopts a two-branch framework for unaligned RGB-NIR guided denoising. (b) The decoder block aggregates aligned guidance features and fuses them with target features. (c) Adaptive sparse guidance propagation transfers reliable coarse candidates to finer scales for efficient sparse matching and guidance aggregation.
Datasets
The model was evaluated on three key datasets:
- FANVID & RGB-NIR Scene: Originally aligned datasets where the authors used 3D Gaussian Splatting (3DGS) to simulate realistic 3D parallax and viewpoint discrepancies.
- MS2: A real-world dataset captured with multi-camera systems, naturally containing cross-modal misalignments.
Key Insights and Ideas
- Mutual Reinforcement: The core "aha" moment is letting restoration and alignment help each other. By treating denoising as a path rather than a destination, the model uses its own intermediate (partially cleaned) results to improve its spatial "vision."
- Plug-and-Play Compatibility: UGD-FM isn't tied to a specific neural architecture. It acts as a wrapper that can be integrated with modern restoration backbones like NAFNet, Restormer, or Uformer.
- Efficiency via Sparsity: By focusing on a few reliable correspondence candidates rather than every possible pixel match, the model handles large-range 2D searches (up to 80-pixel displacements) while remaining computationally feasible.
Key Results
UGD-FM consistently outperformed both single-image denoisers and existing guided denoising methods across all noise levels (Gaussian and Poisson-Gaussian).
- Quantitative Gains: On the FANVID dataset using a Restormer backbone, UGD-FM achieved a PSNR of 30.77 dB, significantly higher than the standard single-image Restormer (29.42 dB) and the previous best guided method, SANet (29.15 dB).
- Speed: The 2-step inference provides a sweet spot, delivering state-of-the-art results while maintaining a reasonable inference time (e.g., ~487ms for a 720p image on a NAFNet backbone).
- Qualitative Quality: As shown in the comparison tables, UGD-FM achieves the best (bold) or second-best (underlined) scores in PSNR, SSIM, and LPIPS across multiple datasets.
Limitations and Future Work
- Severe 3D Parallax: While UGD-FM handles large displacements well, extremely severe 3D parallax involving complex occlusions (where one camera sees something the other doesn't) remains a challenge.
- Computational Overhead: Although sparse propagation is efficient, the two-branch U-Net structure and iterative steps still require more memory and compute than simple single-image models.
- Future Directions: The authors suggest exploring more complex cross-view observations and potentially extending the framework to other guided tasks like thermal image super-resolution.
Summary updated