Upright adjustment with graph convolutional networks

Raehyuk Jung, Sungmin Cho, Junseok KwonICIP 2020

TL;DR

This paper introduces a novel method for automatically correcting the tilt and roll in 360° images by treating them as spherical data. By combining a CNN for feature extraction with a GCN that operates directly on the spherical representation, their approach achieves superior rotation invariance and faster convergence compared to traditional methods.

What Problem Does This Paper Address?

Capturing 360° images, especially without specialized equipment like tripods, often results in images with noticeable camera tilts and rolls. When viewed in virtual reality (VR) using head-mounted displays (HMDs), these distortions can lead to a disorienting and uncomfortable experience, making users feel like they are falling or leaning. The task of "upright adjustment" aims to correct these tilts and rolls, restoring the image to a natural, upright orientation. While existing methods often rely on assumptions about visible features like horizons or project the spherical image onto a 2D grid (losing its inherent spherical nature), this paper proposes a method that directly processes the 360° image in its spherical form.

Methods

The proposed method employs a two-module network:

  1. Convolutional Neural Network (CNN): This module takes the input 360° image (typically in an equirectangular projection) and extracts visual features. Pre-trained architectures like ResNet-18 or DenseNet-121 are used for this purpose.
  2. Graph Convolutional Network (GCN): The key innovation lies here. The feature map from the CNN is converted into a graph representation of the sphere. This involves mapping the feature map to points on the sphere and connecting neighboring nodes (specifically, 6-nearest neighbors) to form the graph. The GCN then processes this graph to predict the position of the "North pole" (the direction opposite to gravity).

A novel loss function, based on the Jensen-Shannon Divergence (JSD), is introduced. This loss function measures the distance between the predicted probability distribution of the North pole and a ground truth distribution, which is generated using a Von Mises-Fisher distribution. This probabilistic approach allows for more robust estimation.

Datasets

The primary dataset used for training and evaluation is the SUN360 dataset. The authors sampled 25,000 images for training, 5,000 for validation, and 4,260 for testing. Images were synthetically rotated to create variations for testing rotation invariance.

Key Insights and Ideas

  • Spherical Processing: The most significant insight is the decision to process 360° images directly in their spherical form using GCNs, rather than forcing them into a 2D grid. This preserves the inherent geometry of the data.
  • GCN for Rotation Invariance: The GCN's ability to operate on graph structures, which are inherently less sensitive to specific orientations than grid-based CNNs, leads to improved rotation invariance. This means the method is less affected by the initial orientation of the input image.
  • Probabilistic North Pole Estimation: Using a probabilistic approach (Von Mises-Fisher distribution and JSD loss) for estimating the North pole is more robust than directly predicting a single point, especially in cases where the horizon is not clearly defined.
  • Faster Convergence: The GCN-based approach converges significantly faster during training compared to methods using fully connected layers, making the training process more efficient.

Key Results

  • Accuracy: The proposed method, using DenseNet-121 with a kappa value of 25, achieved an average error of 4.0° and correctly oriented 97% of images within a 10° tolerance. This outperforms previous methods like Horizon-based algorithms (89.7° average error, 20% within 10°) and Jung et al. [3] (5.9° average error, 96% within 10°), despite being trained for fewer epochs (50 vs. 800).
  • Rotation Invariance: The GCN-based method exhibited better rotation invariance, with a lower standard deviation (STD) of error angles (2.1°) compared to fully connected layer-based methods (4.4°) when tested on randomly rotated images.
  • Convergence Speed: Figure 5 visually demonstrates that the GCN module leads to much faster convergence of the average error during training and validation compared to a fully connected network.

Limitations and Future Work

  • Limitations: While the paper highlights the advantages of GCNs, it's implied that the conversion from a CNN feature map to a graph introduces a step that might lose some fine-grained information. The performance is also dependent on the chosen CNN backbone and the kappa value for the Von Mises-Fisher distribution.
  • Future Work: The authors suggest that their network shows better rotation invariance and faster convergence, implying these are key areas for future exploration and improvement. Further research could explore different graph construction methods or more advanced GCN architectures for spherical data.

Summary updated

BibTeX
@inproceedings{jung2020upright,
  author = {Raehyuk Jung and Sungmin Cho and Junseok Kwon},
  title = {{Upright adjustment with graph convolutional networks}},
  booktitle = {ICIP},
  year = {2020},
}
Figure