Abstract
This paper presents a modular framework for image and video reconstruction tasks, such as super-resolution and denoising, by optimizing the use of correspondences. By decomposing network architectures into distinct modules for representation, alignment, and aggregation, the authors introduce Flow-guided Deformable Alignment (FDA) and Content-conditioned Feature Aggregation (CoFA). This approach achieves state-of-the-art results across multiple benchmarks while maintaining high parameter efficiency.
Introduction
In computer vision, reconstruction tasks—including super-resolution, denoising, and frame interpolation—aim to recover lost textures and details. A primary strategy for high-quality reconstruction is leveraging "supporting" information from external sources, such as adjacent video frames, high-resolution reference photos, or redundant patches within the same image.
The fundamental challenge in this process is establishing accurate correspondences: identifying and aligning similar pixels or patches across different visual contexts. Traditional methods, such as fixed optical flow, often struggle with large displacements or complex motion, while dense deep-learning models can be computationally prohibitive. This research proposes a generic, efficient framework to handle these correspondences across diverse reconstruction tasks.
Methodology
The authors propose a modular pipeline that allows for the independent optimization of three core components:
1. Representation
The research investigates the optimal depth and width for feature extraction. Contrary to the trend of using deep backbones like ResNet, the authors found that shallow feature extractors (fewer layers) are sufficient for capturing the textures necessary for matching, provided the network has enough channels (width).
2. Alignment: Flow-guided Deformable Alignment (FDA)
Alignment is the most critical step in utilizing correspondences. The authors introduce Flow-guided Deformable Alignment (FDA), which combines the strengths of explicit and implicit alignment. FDA uses optical flow as a "coarse" guide to pre-warp features, followed by Deformable Convolution (DCN) to refine the alignment at the feature level. This hybrid approach provides the precision of flow-based methods with the flexibility of DCN.
3. Aggregation: Content-conditioned Feature Aggregation (CoFA)
When multiple supporting images are available (e.g., in burst denoising or multi-frame super-resolution), they must be combined effectively. The proposed CoFA module uses a soft-attention mechanism to weight supporting features based on their similarity to the target, ensuring that only the most relevant information is integrated into the final reconstruction.
Experimental Results
The framework was evaluated across several tasks, including Spatio-Temporal Video Super-Resolution (STVSR), Video Frame Interpolation (VFI), burst denoising, and Reference-based Super-Resolution (RefSR).
Key Findings
- Efficiency: The proposed model for VFI achieved top-tier performance on the Vimeo90K benchmark using only 4% to 24% of the parameters required by competing state-of-the-art (SOTA) models.
- The Impact of Skip Connections: Incorporating long-range skip connections improved PSNR by 1.67 dB. This architectural choice shifts the network's focus from global reconstruction to the recovery of high-frequency details.
- Fine-tuning Flow: While many methods use frozen pre-trained flow estimators, this study demonstrates that "unfreezing" these parameters for task-specific fine-tuning can lead to improvements of up to 1.24 dB.
- Aggregation Performance: The CoFA module outperformed standard averaging or max-pooling methods by over 2 dB in multi-image scenarios.
Conclusion and Future Work
By focusing on a modular design centered on correspondences, the authors demonstrate that high-fidelity reconstruction does not require massive, monolithic architectures. The FDA module provides a stable and precise method for alignment, while CoFA ensures intelligent information fusion.
Future research will address the inherent instability in training deformable modules and explore the application of these modular insights to "blind" restoration tasks, where explicit reference images are unavailable.
Summary updated