TL;DR
GenRF is a generative, registration-free framework that fuses multi-modal images (like infrared and visible) even when they are severely misaligned or degraded. By adapting the FLUX foundation model, it uses cross-attention to "soft-align" features and generative priors to hallucinate missing content, outperforming traditional cascaded pipelines.
What Problem Does This Paper Address?
In the real world, "perfect" image fusion is a myth. When you try to combine a visible-light photo with an infrared (thermal) image, you run into three major roadblocks:
- Geometric Misalignment: Because the sensors are physically separated, they see the world from slightly different angles (parallax).
- Field-of-View (FOV) Mismatch: One camera might see more of the scene than the other, leaving "holes" where data from one modality is missing.
- Image Degradation: Real-world captures are often plagued by low light, noise, or haze, which makes traditional feature matching fail.
Existing solutions usually try a "register-then-fuse" approach. If the registration (alignment) step fails—which it often does with large parallax—the final fused image ends up with "ghosting" artifacts or weird distortions.
Methods
The authors propose GenRF, which skips explicit registration entirely. Instead of trying to warp one image to match the other perfectly, it treats fusion as a conditional generation task.
Key Components:
- Foundation Model Backbone: It builds on FLUX 3, a state-of-the-art Diffusion Transformer (DiT). To make it efficient, they use LoRA (Low-Rank Adaptation) to fine-tune only a small fraction of the weights.
- Cross-Attention as "Soft Alignment": Instead of requiring pixel-to-pixel correspondence, the model uses cross-attention. This allows a pixel in the visible image to "query" relevant thermal information from anywhere in the infrared image, effectively performing a position-agnostic search for complementary data.
- Online Geometric Perturbation: During training, the authors intentionally warp the input images with random affine and elastic deformations. This forces the model to learn how to find correspondences even when things are messy.
- Dynamic Degradation Construction: The model is trained on a wide variety of synthetic corruptions (noise, blur, haze, stripes) so it can perform restoration and fusion simultaneously.
Datasets
- Training: 2,071 image pairs from the DSPFusion dataset, augmented with synthetic warps and degradations.
- Evaluation:
- Real Scene: A newly collected benchmark of 250 pairs from various dual-sensor systems (drones, phones, vehicle-mounted cameras).
- Public Benchmarks: DVTOD (object detection), MSRS, LLVIP, and M3FD.
- Generalization Tasks: Multi-exposure (MEF) and multi-focus (MFF) datasets to test zero-shot capabilities.
Key Insights and Ideas
- Attention is all you need (for alignment): The "aha moment" here is that cross-attention naturally handles misalignment. As shown in Figure 3, the model's attention peaks correctly track objects in the infrared domain even when they are shifted relative to the visible reference.
- Priors fill the gaps: Because it's a generative model, GenRF doesn't just "blend" pixels; it understands scene structure. If the infrared sensor doesn't cover a certain area, the model uses its learned priors to infer what the thermal signature should look like, avoiding the ugly black borders or padding seen in other methods.
- Joint Optimization > Cascaded Steps: By handling restoration, alignment, and fusion in one pass, the model avoids the error accumulation that kills traditional pipelines.
Key Results
GenRF sets a new state-of-the-art across multiple metrics:
- Visual Quality: It achieves the highest scores in perceptual metrics like DeQA (3.213) and QualiCLIP (0.708) on real-world scenes, significantly beating methods that rely on explicit registration.
- Downstream Performance: On the DVTOD object detection task, using GenRF-fused images improved the mAP50:95 of a YOLOv11 detector to 0.524, compared to 0.494 for the next best method.
- Zero-Shot Generalization: Without any task-specific fine-tuning, GenRF successfully fused multi-exposure and multi-focus images, proving that its learned fusion logic is remarkably general.
Limitations and Future Work
The primary drawback is computational speed. Because GenRF relies on an iterative diffusion process (30 denoising steps), it takes about 24 seconds to fuse a single 640x480 image pair on an RTX 4090. This is far from real-time.
The authors suggest that future work could explore one-step generative fusion (using techniques like Consistency Models or adversarial distillation) to bring the latency down for mobile or robotics applications. Additionally, while the model is robust, it is still a "heavy" foundation model; distilling this knowledge into a smaller, more efficient architecture would be a logical next step.
Summary updated