Abstract
Omnidirectional images (ODIs) are typically captured at low resolutions due to constraints in collection, storage, and transmission. Traditional 2D image super-resolution (SR) methods struggle with spherical ODIs because of non-uniform pixel density and varying texture complexity across latitudes. We propose LAU-Net, a novel latitude adaptive upscaling network that allows different latitudes to adopt distinct upscaling factors. By employing a Laplacian multi-level separation architecture and a deep reinforcement learning scheme with a latitude-adaptive reward, LAU-Net automatically selects optimal upscaling factors for different latitude bands. Extensive results demonstrate that LAU-Net significantly advances ODI super-resolution performance.
Introduction: The Challenge of Spherical Projections
Omnidirectional images (360° photos) are commonly stored using Equirectangular Projection (ERP). While convenient for 2D storage, ERP introduces significant geometric distortion. Areas near the "equator" (low latitudes) are dense and rich in texture, whereas the "poles" (high latitudes) are stretched and over-sampled.
Standard 2D super-resolution methods treat every pixel uniformly, which leads to two major inefficiencies:
- Computational Waste: Excessive processing is spent on polar regions that are already over-sampled due to projection stretching.
- Detail Deficiency: Insufficient focus is given to the equator, where texture is most complex and human attention is typically centered in VR environments.
Methodology: Latitude Adaptive Upscaling
The proposed Latitude Adaptive Upscaling Network (LAU-Net) addresses these issues by treating super-resolution as a progressive, multi-level task tailored to local image characteristics.
Multi-level Pyramid and Band Splitting
The network utilizes a three-level architecture for 8x upscaling. Each level integrates a Channel Attention Dense Net (CAD-net) for deep feature extraction and a Spatial Segmentation Module (SSM) that divides the ERP image into horizontal latitude bands.
Reinforcement Learning "Early Quit" Strategy
The core innovation is a deep reinforcement learning (RL) agent that acts as an evaluator for each latitude band. At each level, the agent decides whether a patch is "easy" (high latitude, low complexity) or "hard" (low latitude, high complexity).
- Easy patches exit the network early after the first level of upscaling, saving significant computational resources.
- Hard patches proceed through deeper levels to achieve maximum detail reconstruction.
The RL agent is trained with a Latitude-Adaptive Reward function that balances reconstruction accuracy (PSNR) against computational cost (FLOPs), specifically weighted to prioritize the visually critical low-latitude regions.
Experimental Results
The authors evaluated LAU-Net on a new ODI-SR Database containing 1,000 high-quality 360° images and the standard SUN 360 benchmark.
Quantitative and Qualitative Superiority
LAU-Net outperforms state-of-the-art (SOTA) methods like RCAN, EDSR, and EBRN. On the ODI-SR dataset (8x scale), it achieved a WS-PSNR of 24.36 dB, surpassing the next best method by 0.07 dB and standard Bicubic interpolation by nearly 5 dB. Qualitatively, LAU-Net recovers sharper edges and finer textures in equatorial regions where other methods produce blur or aliasing.
Efficiency Gains
By allowing polar regions to exit the network early, LAU-Net reduces computational overhead by approximately 45% FLOPs compared to non-adaptive models of similar depth. It achieves an inference time of 0.352s, which is significantly faster than EDSR (2.231s) and RCAN (0.416s).
Conclusion
LAU-Net represents the first attempt to incorporate latitude-specific characteristics into omnidirectional image super-resolution. By leveraging reinforcement learning for adaptive upscaling, it achieves a superior balance between visual quality and computational efficiency. Future work may extend this latitude-aware approach to omnidirectional video, where temporal redundancies offer further opportunities for optimization.
Summary updated