Grid Based Spherical CNN for Object Detection from Panoramic Images

Dawen Yu, Shunping Ji

TL;DR

This paper introduces the Grid-Based Spherical CNN (G-SCNN), a novel approach for object detection in panoramic images. By transforming spherical images into a grid map and carefully scaling object features, G-SCNN overcomes the bandwidth limitations and location ambiguity issues that previously hindered spherical CNNs for detection tasks, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Traditional Convolutional Neural Networks (CNNs) excel at object detection in planar images due to their inherent translation invariance. However, applying these to spherical images, common in applications like autonomous driving and virtual reality, is challenging. Spherical CNNs (SCNNs) have shown promise for classification on spheres, but two key issues prevent their direct use for object detection:

  1. Loss of Location Information: Spherical convolutions, operating on the entire sphere, make it difficult to precisely localize objects and retrieve their bounding boxes.
  2. Bandwidth Requirements: To preserve the detail of small objects on a sphere, SCNNs require very high bandwidth (resolution), leading to computationally prohibitive memory and processing demands. This is especially problematic for object detection where objects might occupy only a small portion of the entire spherical image.

Methods

The authors propose the Grid-Based Spherical CNN (G-SCNN) to address these challenges. The core innovations are:

  • Conformal Grid Map Transformation: Instead of processing the raw spherical image, the input is transformed into a conformal grid map. This allows for a more manageable input representation for the SCNN.
  • Object Scaling on Grid Map: Crucially, the bounding box of a detected object is scaled to occupy a significant portion of this grid map (e.g., 7x7 on a 28x28 grid). This ensures that the object's features are not overly compressed and are effectively captured by the subsequent spherical convolutions, mitigating the bandwidth problem.
  • Rotation-Augmented Planar Region Proposal Network (RPN): To address the loss of location information, the paper utilizes a standard planar RPN for initial bounding box proposals. This RPN is augmented with a data augmentation strategy that increases its rotation invariance, helping to better identify potential object locations across different orientations.
  • G-SCNN Architecture: The G-SCNN itself consists of an S2 convolution followed by an SO(3) convolution, which are known to capture spherical and rotation-invariant features, respectively. This is then followed by fully connected layers for classification and bounding box regression.

Datasets

The paper introduces a new dataset:

  • WHU (Wuhan University) Panoramic Dataset: This dataset comprises 600 street-view panoramic images captured by a vehicle-borne panoramic camera. It includes 5636 annotated objects of interest across four classes: light, crosswalk, crosswalk warning line, and car. The images are provided in a Driscoll-Healy projection format.

The authors note that existing datasets are often simulated or indoor-based, making this real-world street-scene dataset a valuable contribution.

Key Insights and Ideas

  • Grid Transformation is Key: The simple yet effective idea of transforming the spherical image into a grid map and then scaling object features to occupy a substantial part of this grid is the core innovation. This elegantly solves the bandwidth issue without sacrificing object information.
  • Separation of Concerns: The G-SCNN effectively separates the initial object proposal (handled by a planar RPN with rotation augmentation) from the detailed feature extraction and classification (handled by the spherical convolutions). This modular approach leverages the strengths of both planar and spherical CNNs.
  • SCNNs are Not Directly Detectors: The paper clearly demonstrates why original SCNNs, designed for classification, struggle with object detection due to their inherent limitations in localization and bandwidth management.
  • Real-World Data Matters: The creation of the WHU dataset highlights the need for real-world, diverse panoramic image datasets for robust evaluation of object detection algorithms in complex environments.

Key Results

  • Superior Performance: The G-SCNN significantly outperforms the original SCNN for object detection, achieving much higher mAP (mean average precision) scores.
  • Competitive with State-of-the-Art: The proposed G-SCNN achieves competitive or superior results compared to mainstream planar object detection methods like Faster R-CNN and SSD on their custom panoramic dataset. For instance, on mAP50, the G-SCNN achieved 60.0%, outperforming Faster R-CNN (57.9%) and SSD (56.7%).
  • Improved Accuracy: The G-SCNN also shows better bounding box accuracy, as indicated by higher mAP75 scores.
  • Rotation Invariance: Data augmentation with rotation sampling further improves the performance of G-SCNN, demonstrating its robustness to varying object orientations.
  • Efficiency Trade-off: While G-SCNN is slower than planar CNNs due to spherical convolutions, its efficiency is comparable to other spherical CNN methods.

Limitations and Future Work

  • Reliance on Planar RPN: The current G-SCNN still relies on a planar RPN for initial bounding box proposals. The authors suggest that future work could explore more sophisticated algorithms for directly locating bounding boxes within the SO(3) space, eliminating the need for a planar RPN.
  • Feature Localization in SO(3): The paper notes that while spherical convolutions capture features on the sphere, accurately localizing these features in the SO(3) space remains an open challenge. They hope to discover effective ways to achieve this in future research.
  • Encoder Impact: While VGG16 was found to be the best encoder for the RPN in terms of accuracy and efficiency, further exploration of different encoder architectures could potentially yield further improvements.

Summary updated

Abstract

Recently proposed spherical convolutional neural networks (SCNNs) have shown advantages over conventional planar CNNs on classifying spherical images. However, two factors hamper their application in an objection detection task. First, a convolution in S2 (a two-dimensional sphere in three-dimensional space) or SO(3) (three-dimensional special orthogonal group) space results in the loss of an object’s location. Second, overlarge bandwidth is required to preserve a small object’s information on a sphere because the S2/SO(3) convolution must be performed on the whole sphere, instead of a local image patch. In this study, we propose a novel grid-based spherical CNN (G-SCNN) for detecting objects from spherical images. According to input bandwidth, a sphere image is transformed to a conformal grid map to be the input of the S2/SO3 convolution, and an object’s bounding box is scaled to cover an adequate area of the grid map. This solves the second problem. For the first problem, we utilize a planar region proposal network (RPN) with a data augmentation strategy that increases rotation invariance. We have also created a dataset including 600 street view panoramic images captured from a vehicle-borne panoramic camera. The dataset contains 5636 objects of interest annotated with class and bounding box and is named as WHU (Wuhan University) panoramic dataset. Results on the dataset proved our grid-based method is extremely better than the original SCNN in detecting objects from spherical images, and it outperformed several mainstream object detection networks, such as Faster R-CNN and SSD.
BibTeX
@inproceedings{yu2019grid,
  author = {Dawen Yu and Shunping Ji},
  title = {{Grid Based Spherical CNN for Object Detection from Panoramic Images}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=Eu8_2TqUxn2},
}

Loading related content...

Figure