360-Indoor: Towards Learning Real-World Objects in 360° Indoor Equirectangular Images

Shih-Han Chou, Cheng Sun, Wen-Yen Chang, Wan Ting Hsu, Min Sun, Jianlong Fu

Abstract

While there are several widely used object detection datasets, current computer vision algorithms are still limited in conventional images. Such images narrow our vision in a restricted region. On the other hand, 360° images provide a thorough sight. In this paper, our goal is to provide a standard dataset to facilitate the vision and machine learning communities in the 360° domain. To facilitate the research, we present a real-world 360° panoramic object detection dataset, 360-Indoor, which is a new benchmark for visual object detection and class recognition in 360° indoor images. It is achieved by gathering images of complex indoor scenes containing common objects and the intensive annotated bounding field-of-view. In addition, 360-Indoor has several distinct properties: (1) the largest category number (37 labels in total); (2) the most complete annotations on average (27 bounding boxes per image). The selected 37 objects are all common in indoor scenes. With around 3k images and 90k labels in total, 360-Indoor achieves the largest dataset for detection in 360° images. In the end, extensive experiments on the state-of-the-art methods for both classification and detection are provided.

Introduction

Conventional computer vision research primarily focuses on images with a narrow field of view. However, the rise of 360° cameras in robotics, virtual reality, and real estate has created a demand for models capable of processing panoramic data. The primary obstacle is geometric distortion; 360° images are typically represented via equirectangular projection, which causes significant warping near the poles. Standard models trained on datasets like MSCOCO struggle with these distortions.

Example of a complex indoor scene (church) in the 360-Indoor dataset, showcasing high-density annotations for windows, lights, and chairs.

The 360-Indoor dataset addresses this gap by providing a large-scale, diverse, and real-world benchmark specifically for the 360° domain.

The 360-Indoor Dataset

The 360-Indoor dataset is the largest of its kind, comprising 3,335 images and 89,148 annotated objects. It features 37 object categories—such as chairs, lights, windows, and appliances—grouped into five super-categories.

Overview of the 360-Indoor dataset diversity, featuring various indoor environments like bathrooms, living rooms, and hallways with BFoV annotations.

A defining characteristic of the dataset is its annotation density, averaging 27 bounding boxes per image. Images were sourced from platforms like Flickr and Kuula, ensuring a wide variety of real-world indoor environments.

Methodology

To accommodate spherical geometry, the researchers utilized Bounding Field-of-Views (BFoVs) instead of standard 2D bounding boxes. A BFoV is defined by latitude/longitude coordinates ($\phi, \theta$) and field-of-view dimensions ($h, w$). This approach accounts for "circular padding," where objects may wrap around the image edges.

The study benchmarked several state-of-the-art architectures:

  • YOLOv3
  • Faster R-CNN
  • Feature Pyramid Network (FPN)
  • SphereNet: A specialized architecture using spherical convolutions to handle equirectangular distortion.

Results and Discussion

Experimental results revealed several key insights:

  1. Top Performance: FPN with a ResNet101 backbone achieved the highest accuracy with a 33.6% mean Average Precision (mAP).
  2. The Pre-training Paradox: Standard models pre-trained on massive conventional datasets (ImageNet/COCO) outperformed specialized spherical models like SphereNet trained from scratch. This suggests that current spherical architectures lack the "feature wealth" provided by large-scale conventional pre-training.
  3. Anchor Optimization: For YOLOv3, using k-means clustering to generate custom anchor boxes from the 360-Indoor dataset significantly improved performance compared to default COCO anchors.
  4. Distortion Robustness: While most objects reside in low-distortion regions ($\pm 30^\circ$ latitude), the dataset's inclusion of objects in high-distortion zones ($\pm 60^\circ$ to $\pm 90^\circ$) provides a rigorous test for model robustness.

Conclusion

360-Indoor establishes a challenging new benchmark for the computer vision community. While current models achieve a peak mAP of 33.6%, there is significant room for improvement. Future research may focus on leveraging global context—utilizing the full 360° view to infer object relationships—and developing large-scale spherical pre-training datasets to further enhance model performance in panoramic environments.

Summary updated

Abstract

While there are several widely used object detection datasets, current computer vision algorithms are still limited in conventional images. Such images narrow our vision in a restricted region. On the other hand, 360{\deg} images provide a thorough sight. In this paper, our goal is to provide a standard dataset to facilitate the vision and machine learning communities in 360{\deg} domain. To facilitate the research, we present a real-world 360{\deg} panoramic object detection dataset, 360-Indoor, which is a new benchmark for visual object detection and class recognition in 360{\deg} indoor images. It is achieved by gathering images of complex indoor scenes containing common objects and the intensive annotated bounding field-of-view. In addition, 360-Indoor has several distinct properties: (1) the largest category number (37 labels in total). (2) the most complete annotations on average (27 bounding boxes per image). The selected 37 objects are all common in indoor scene. With around 3k images and 90k labels in total, 360-Indoor achieves the largest dataset for detection in 360{\deg} images. In the end, extensive experiments on the state-of-the-art methods for both classification and detection are provided. We will release this dataset in the near future.
BibTeX
@inproceedings{chou2020indoor,
  author = {Shih-Han Chou and Cheng Sun and Wen-Yen Chang and Wan Ting Hsu and Min Sun and Jianlong Fu},
  title = {{360-Indoor: Towards Learning Real-World Objects in 360° Indoor Equirectangular Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=25CkdRfb6Zv},
}
Figure