360-Indoor: Towards Learning Real-World Objects in 360deg Indoor Equirectangular Images

Shih-Han Chou, Cheng Sun, Wen-Yen Chang, Wan-Ting Hsu, Min Sun, Jianlong FuWACV 2020

TL;DR

This paper introduces 360-Indoor, the first large-scale, real-world dataset for object detection in 360° indoor equirectangular images. It addresses the significant distortion challenges in 360° imagery and provides a benchmark to advance object detection capabilities in this increasingly prevalent format.

What Problem Does This Paper Address?

Current object detection research heavily relies on standard, narrow field-of-view (NFoV) images. While effective for conventional cameras, these datasets fail to capture the immersive, all-encompassing nature of 360° imagery, which is becoming ubiquitous in applications like robotics, virtual reality, and content creation. A major hurdle in leveraging 360° images for object detection is the inherent geometric distortion, particularly severe at the poles of equirectangular projections. Existing datasets either focus on 2D projections, synthetic data, or lack the scale and diversity needed to train robust 360° object detectors. This paper aims to bridge this gap by providing a comprehensive, real-world dataset and evaluating existing methods on it.

Methods

The core contribution is the 360-Indoor dataset itself. Key aspects of its creation and the evaluation methods include:

  • Bounding Field-of-Views (BFoVs): Instead of traditional bounding boxes, the dataset uses BFoVs, which are better suited to represent objects in distorted 360° images. A BFoV is defined by a viewpoint and the width and height of the field-of-view, allowing for more accurate annotation of objects regardless of their position in the equirectangular projection.
  • Annotation Tool: A custom tool was developed to facilitate the annotation of BFoVs, allowing annotators to select object centers and adjust the field-of-view dimensions.
  • Category Selection: The authors curated 37 object categories, drawing from COCO but adapting them for indoor scenes, merging similar items (e.g., 'lamp' and 'light'), and adding common indoor objects (e.g., 'washer', 'heater').
  • Image Collection: Images were sourced from Flickr, Kuula, and the Narrated 360° videos dataset, with careful filtering to ensure real-world indoor scenes and deduplication.
  • Evaluation of Standard Detectors: The paper evaluates three prominent object detection architectures:
  • One-stage: YOLOv3
    • Two-stage: Faster R-CNN and FPN (Feature Pyramid Network)
  • Spherical CNNs: The authors also investigate the use of SphereNet, a spherical convolutional neural network designed to handle 360° distortions, by replacing standard convolutional layers with spherical equivalents.
  • BFoV to Bounding Box Transformation: For compatibility with existing detectors, a method is proposed to transform BFoV annotations into conventional bounding boxes by projecting ground truth annotations onto the tangent plane.

Datasets

  • 360-Indoor Dataset (New Benchmark):
    • Content: 3,335 real-world indoor equirectangular images.
  • Annotations: 89,148 Bounding Field-of-Views (BFoVs) across 37 object categories.
  • Key Features: Largest category number (37) and most annotations per image (27 on average) for 360° object detection.
  • Existing Datasets (for comparison/context):
    • COCO: Used as a baseline for pre-trained models and for calculating overlap-based mAP.
  • PASCAL VOC: Mentioned as a foundational object detection dataset.
  • Other 360° Datasets (Table 1): Pano2Vid, Sports-360, Narrated-360, Wild-360, SUN360, ERA, FlyingCars. These are discussed to highlight the novelty and scale of 360-Indoor.

Key Insights and Ideas

  • Distortion is a Major Hurdle: The paper strongly emphasizes that standard object detection methods struggle with the geometric distortions inherent in 360° equirectangular images, especially in polar regions.
  • BFoVs are Crucial: The use of Bounding Field-of-Views is a necessary adaptation for accurate annotation and potentially for developing specialized 360° detectors.
  • Real-World Data is Key: Training on a real-world 360° dataset like 360-Indoor significantly boosts performance compared to using models pre-trained solely on conventional images or even synthetic 360° data.
  • Anchor Box Optimization Matters: For models like YOLOv3, which are sensitive to anchor box configurations, calculating anchor boxes specifically from the 360-Indoor dataset is critical for optimal performance.
  • Pre-trained Standard CNNs Outperform Scratch Spherical CNNs (Initially): While SphereNet is designed for 360° distortions, the paper found that standard CNNs pre-trained on ImageNet, when adapted, performed better than SphereNet trained from scratch. This highlights the importance of large-scale pre-training and suggests that specialized 360° detectors might need further development or better pre-training strategies.

Key Results

  • Significant Performance Gains: All evaluated object detection models (YOLOv3, Faster R-CNN, FPN) trained on the 360-Indoor dataset significantly outperform their COCO-pretrained baselines when tested on 360-Indoor images.
  • FPN Leads: FPN achieved the best performance, reaching 33.6% mAP, demonstrating its robustness in handling distortions and small objects in 360° images.
  • YOLOv3 Sensitivity: YOLOv3's performance was highly dependent on anchor box settings, with custom anchors derived from 360-Indoor yielding better results than COCO anchors.
  • SphereNet vs. Standard CNNs: Faster R-CNN with standard Conv2d (pre-trained) achieved 30.2% mAP, while the same architecture with SphereConv2d trained from scratch achieved 21.7% mAP. This suggests that pre-training is currently more impactful than specialized spherical convolutions trained from scratch.

Limitations and Future Work

  • Limitations:
    • The paper notes that even with the best models, the 360-Indoor dataset is not yet saturated, indicating room for improvement.
  • The initial evaluation of SphereNet suggests that while promising, it requires further development, particularly regarding pre-training strategies, to outperform adapted standard CNNs.
  • Future Work:
    • Developing dedicated object detectors specifically designed to overcome image distortion and leverage the full contextual information available in 360° scenes.
  • Exploring more advanced techniques for handling the unique challenges of 360° imagery, potentially combining spherical convolutions with robust pre-training.
  • Expanding the dataset or creating similar benchmarks for other 360° domains (e.g., outdoor scenes).

Summary updated

BibTeX
@inproceedings{chou2020indoor,
  author = {Shih-Han Chou and Cheng Sun and Wen-Yen Chang and Wan-Ting Hsu and Min Sun and Jianlong Fu},
  title = {{360-Indoor: Towards Learning Real-World Objects in 360deg Indoor Equirectangular Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure