LAU-Net: Latitude Adaptive Upscaling Network for Omnidirectional Image Super-Resolution

Xin Deng, Hao Wang, Mai Xu, Yichen Guo, Yuhang Song, Li YangCVPR 2021 · OpenReview

Abstract

Omnidirectional images (ODIs) are typically captured at low resolutions due to constraints in collection, storage, and transmission. Traditional 2D image super-resolution (SR) methods struggle with spherical ODIs because of non-uniform pixel density and varying texture complexity across latitudes. We propose LAU-Net, a novel latitude adaptive upscaling network that allows different latitudes to adopt distinct upscaling factors. By employing a Laplacian multi-level separation architecture and a deep reinforcement learning scheme with a latitude-adaptive reward, LAU-Net automatically selects optimal upscaling factors for different latitude bands. Extensive results demonstrate that LAU-Net significantly advances ODI super-resolution performance.

Introduction: The Challenge of Spherical Projections

Omnidirectional images (360° photos) are commonly stored using Equirectangular Projection (ERP). While convenient for 2D storage, ERP introduces significant geometric distortion. Areas near the "equator" (low latitudes) are dense and rich in texture, whereas the "poles" (high latitudes) are stretched and over-sampled.

Standard 2D super-resolution methods treat every pixel uniformly, which leads to two major inefficiencies:

  1. Computational Waste: Excessive processing is spent on polar regions that are already over-sampled due to projection stretching.
  2. Detail Deficiency: Insufficient focus is given to the equator, where texture is most complex and human attention is typically centered in VR environments.
Figure 6: Performance degradation of standard SR methods at equatorial latitudes (0°), highlighting the need for latitude-aware processing.

Methodology: Latitude Adaptive Upscaling

The proposed Latitude Adaptive Upscaling Network (LAU-Net) addresses these issues by treating super-resolution as a progressive, multi-level task tailored to local image characteristics.

Figure 1: The LAU-Net architecture uses a multi-level pyramid to adaptively upscale different latitude bands based on feature complexity and latitude.

Multi-level Pyramid and Band Splitting

The network utilizes a three-level architecture for 8x upscaling. Each level integrates a Channel Attention Dense Net (CAD-net) for deep feature extraction and a Spatial Segmentation Module (SSM) that divides the ERP image into horizontal latitude bands.

Reinforcement Learning "Early Quit" Strategy

The core innovation is a deep reinforcement learning (RL) agent that acts as an evaluator for each latitude band. At each level, the agent decides whether a patch is "easy" (high latitude, low complexity) or "hard" (low latitude, high complexity).

  • Easy patches exit the network early after the first level of upscaling, saving significant computational resources.
  • Hard patches proceed through deeper levels to achieve maximum detail reconstruction.

The RL agent is trained with a Latitude-Adaptive Reward function that balances reconstruction accuracy (PSNR) against computational cost (FLOPs), specifically weighted to prioritize the visually critical low-latitude regions.

Experimental Results

The authors evaluated LAU-Net on a new ODI-SR Database containing 1,000 high-quality 360° images and the standard SUN 360 benchmark.

Quantitative and Qualitative Superiority

LAU-Net outperforms state-of-the-art (SOTA) methods like RCAN, EDSR, and EBRN. On the ODI-SR dataset (8x scale), it achieved a WS-PSNR of 24.36 dB, surpassing the next best method by 0.07 dB and standard Bicubic interpolation by nearly 5 dB. Qualitatively, LAU-Net recovers sharper edges and finer textures in equatorial regions where other methods produce blur or aliasing.

Efficiency Gains

By allowing polar regions to exit the network early, LAU-Net reduces computational overhead by approximately 45% FLOPs compared to non-adaptive models of similar depth. It achieves an inference time of 0.352s, which is significantly faster than EDSR (2.231s) and RCAN (0.416s).

Conclusion

LAU-Net represents the first attempt to incorporate latitude-specific characteristics into omnidirectional image super-resolution. By leveraging reinforcement learning for adaptive upscaling, it achieves a superior balance between visual quality and computational efficiency. Future work may extend this latitude-aware approach to omnidirectional video, where temporal redundancies offer further opportunities for optimization.

Summary updated

Abstract

The omnidirectional images (ODIs) are usually at low-resolution, due to the constraints of collection, storage and transmission. The traditional two-dimensional (2D) image super-resolution methods are not effective for spherical ODIs, because ODIs tend to have non-uniformly distributed pixel density and varying texture complexity across latitudes. In this work, we propose a novel latitude adaptive upscaling network (LAU-Net) for ODI super-resolution, which allows pixels at different latitudes to adopt distinct upscaling factors. Specifically, we introduce a Laplacian multi-level separation architecture to split an ODI into different latitude bands, and hierarchically upscale them with different factors. In addition, we propose a deep reinforcement learning scheme with a latitude adaptive reward, in order to automatically select optimal upscaling factors for different latitude bands. To the best of our knowledge, LAU-Net is the first attempt to consider the latitude difference for ODI super-resolution. Extensive results demonstrate that our LAU-Net significantly advances the super-resolution performance for ODIs.
BibTeX
@inproceedings{deng2021lau,
  author = {Xin Deng and Hao Wang and Mai Xu and Yichen Guo and Yuhang Song and Li Yang},
  title = {{LAU-Net: Latitude Adaptive Upscaling Network for Omnidirectional Image Super-Resolution}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=JuYzzuIGLPR},
}
Figure