TL;DR
This paper introduces OHT, a novel hybrid Transformer network for light field super-resolution that effectively tackles occlusions by incorporating occlusion maps. By combining spatial-angular separable convolutions (SASep-Conv) and angular self-attention (ASA), OHT achieves superior performance in reconstructing high-resolution light fields, outperforming existing methods.
What Problem Does This Paper Address?
Light field imaging captures both spatial and angular information, offering richer scene representations than traditional 2D images. However, commercial light field cameras often have limited spatial resolution, necessitating super-resolution (SR) techniques to enhance image quality. While deep learning methods, particularly those based on Transformers, have shown promise, they struggle to efficiently capture both global and local spatial-angular correlations. A significant challenge, especially in real-world scenarios, is the presence of occlusions. Existing methods often overlook this geometric information, leading to performance degradation and artifacts in the reconstructed images. This paper aims to address these limitations by developing a light field SR method that explicitly accounts for occlusions and efficiently models both local and global dependencies.
Methods
The proposed Occlusion-Embedded Hybrid Transformer (OHT) is a hybrid network that integrates convolutional and Transformer architectures.
- Occlusion Map Generator: OHT first generates occlusion maps for each view. This is achieved by leveraging the photometric consistency prior and a coarse disparity map estimated using OACCNet. These maps highlight regions where pixels are occluded or inconsistent across views.
- Occlusion-Embedded Mix Layer: The generated occlusion maps are then fused with initial features extracted from the low-resolution (LR) input. This fusion is performed using an "occlusion-embedded mix layer" which helps the network prioritize information from non-occluded regions.
- Spatial-Angular Separable Convolution (SASep-Conv): To efficiently capture spatial-angular correlations, OHT introduces SASep-Conv. This module decouples 3D convolutions into parallel spatial and angular branches, offering a more lightweight and effective alternative to standard 3D convolutions.
- Angular Self-Attention (ASA): Instead of traditional spatial self-attention, OHT employs ASA, which operates along the angular dimension. This mechanism is designed to capture long-range angular correlations efficiently and is less computationally intensive than spatial attention for light field data.
- Hybrid Blocks and Convolutional Feed-Forward Layer (CFFL): The core of the network consists of "hybrid blocks" that leverage the SASep-Conv and ASA. A CFFL is also incorporated to further refine features and focus on informative regions.
Datasets
The paper uses the BASiCLFSR benchmark for training and evaluation. This benchmark comprises 144 training scenes and 23 test scenes from five light field datasets: HCI-new, HCI-old, EPFL, INRIA, and STFGantry. The authors extract the central 5x5 views from each light field for their experiments.
Key Insights and Ideas
- Occlusions are critical: The paper strongly emphasizes that ignoring occlusions significantly degrades light field SR performance. Explicitly modeling and utilizing occlusion information is crucial for achieving state-of-the-art results.
- Angular attention is key: Traditional self-attention mechanisms are often computationally expensive and not optimally suited for the structured nature of light fields. The proposed Angular Self-Attention (ASA) effectively leverages angular correlations, offering a more efficient and effective way to capture long-range dependencies.
- Hybrid approach is powerful: Combining the local inductive bias of convolutions (via SASep-Conv) with the global dependency modeling of Transformers (via ASA) creates a synergistic effect, leading to superior performance.
- SASep-Conv offers efficiency: The proposed SASep-Conv provides a more efficient way to extract spatial-angular features compared to standard 3D convolutions, which is vital for handling high-dimensional light field data.
Key Results
- Quantitative Performance: OHT-B (the base version) achieves the highest PSNR and SSIM values on the BASiCLFSR benchmark for both ×2 and ×4 SR tasks, outperforming several state-of-the-art methods. For example, OHT-B outperforms DET by 0.08 dB in PSNR for ×2 SR.
- Qualitative Performance: Visual results demonstrate that OHT generates sharper and more precise reconstructions with fewer artifacts, especially in occluded regions, compared to other methods. Ablation studies confirm the effectiveness of each proposed component (occlusion embedding, SASep-Conv, ASA, CFFL).
Limitations and Future Work
- Limitations: While OHT achieves superior performance, the paper notes that the complexity of the occlusion map generation and the overall model size (especially for OHT-B) are considerations. The authors also mention that accurately estimating occlusion maps is a non-trivial task.
- Future Work: The authors suggest future research directions including optimizing the model's complexity and exploring its applications in related tasks such as light field image denoising and reflection removal.
Summary updated