Generation of Upright Panoramic Image From Non-Upright Panoramic Image

Jingguo Liu, Heyu Chen, Shigang Li, Jianfeng LiWACV 2024

TL;DR

This paper introduces a novel end-to-end deep learning approach for automatically generating upright panoramic images from non-upright ones. By formulating the problem as a direct pixelwise image-to-image mapping, their method bypasses traditional multi-step processes of inclination estimation and resampling, achieving significantly faster processing speeds and a more compact model.

What Problem Does This Paper Address?

Spherical cameras, often used for capturing panoramic views, can be tilted, resulting in "non-upright" images. Correcting this tilt to produce an "upright" image is crucial for many applications, including virtual reality, augmented reality, and scene understanding. Traditional methods tackle this by first estimating the camera's inclination angles (pitch and roll) and then resampling the image to compensate for this tilt. However, accurately estimating these angles, especially in complex or natural scenes, is challenging. Furthermore, the resampling process, while often accelerated using lookup tables (LUTs), requires significant memory to store these tables, which are dependent on the estimated angles. This paper aims to overcome these limitations by proposing a direct, single-stage method for upright panoramic image generation.

Methods

The core of this paper's contribution lies in reframing the upright adjustment problem as a pixelwise image-to-image mapping. Instead of estimating inclination angles and then resampling, the authors propose an end-to-end neural network that directly learns to transform a non-upright panoramic image into its upright counterpart.

The proposed network consists of two main components:

  • Encoder: This part of the network is responsible for extracting "inclination features" from the input non-upright panoramic image. The authors emphasize the use of double-convolution layers in the encoder, arguing that single convolutions are insufficient for capturing the necessary features. This encoder progressively downsamples the input image to obtain a compact representation of these inclination features.
  • Decoder: This component takes the inclination features from the encoder and estimates the pixel displacement required to generate the upright image. The decoder upsamples the features and outputs a 2-channel displacement map, where each pixel indicates how much it needs to shift horizontally and vertically to reach its upright position. This displacement map is then used with a grid_sample function to warp the input image into the final upright panoramic image.

The network is trained using a combination of L1 loss and perceptual loss (derived from VGG19 features). L1 loss encourages pixel-wise similarity to the ground truth, while perceptual loss helps preserve high-frequency details and improve overall image quality.

Datasets

The primary dataset used for training and evaluation is Matterport3D. To create non-upright panoramic images for training, the authors artificially rotated the upright images from Matterport3D by random angles in the range of [-90°, 90°] for both pitch and roll. They used 70% of the data for training, 15% for evaluation, and 15% for testing. To further expand the dataset and cover various angles, five different random rotations were applied to each image after the initial split.

The Stanford2D3D dataset was also used for evaluation, demonstrating the method's generalization capabilities.

Key Insights and Ideas

  • Problem Reframing: The most significant insight is treating upright adjustment as a direct pixelwise mapping problem rather than a two-stage process of estimation and resampling. This shift simplifies the pipeline and opens up possibilities for end-to-end learning.
  • End-to-End Learning for Inclination: Instead of explicitly predicting inclination angles, the network learns to extract "inclination features" implicitly. This is argued to be more robust, especially in complex scenes where explicit angle estimation is difficult.
  • Pixel Displacement as the Core Output: The network's direct output is a pixel displacement map, which is a more fundamental representation of the transformation needed for upright adjustment.
  • Efficiency through Direct Mapping: By avoiding intermediate steps like LUT generation and explicit angle calculation, the proposed method achieves remarkable speed and reduced model size.

Key Results

  • Speed: The proposed method achieves a processing speed of 9ms per frame for a 256x512 pixel panoramic image, significantly faster than traditional methods.
  • Model Size: The trained model is lightweight, requiring only 163MB of storage.
  • Accuracy: The method demonstrates competitive accuracy, with NRMSE and NMAE values between 1° and 2° when evaluated against ground truth.
  • Image Quality: Qualitative results show that the generated upright images are visually comparable to LUT-based methods, with a lower FID score (31.21) compared to LUT (41.7467) and Chen et al. [5] (59.97), indicating better image quality.
  • Application Robustness: The method shows robustness in downstream tasks like pedestrian detection and depth estimation, even when applied to outdoor images or when the original images are distorted.

Limitations and Future Work

  • Resolution Dependence: The current network is trained for a specific resolution (256x512). While the authors believe the idea can be extended to higher resolutions, it would likely require retraining or architectural adjustments.
  • Future Work: The authors suggest that improving the quality of generated images is a direction for future work. They also mention that the spherical-based projection can reach real-time performance on GPUs with high-level optimization, but their method offers a competitive alternative by avoiding complex pre-processing.

Summary updated

BibTeX
@inproceedings{liu2024generation,
  author = {Jingguo Liu and Heyu Chen and Shigang Li and Jianfeng Li},
  title = {{Generation of Upright Panoramic Image From Non-Upright Panoramic Image}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}

Loading related content...

Figure