Visual Question Answering on 360° Images

Shih-Han Chou, Wei-Lun Chao, Wei-Sheng Lai, Min Sun

TL;DR

This paper introduces VQA 360°, a novel task and dataset for visual question answering on 360° images, which present unique challenges due to their expansive field of view and inherent spatial distortion. They propose a cubemap-based model that effectively handles this distortion and spatial reasoning, achieving state-of-the-art results on their new benchmark.

What Problem Does This Paper Address?

Visual Question Answering (VQA) has seen significant progress, but existing work is largely confined to standard "normal field-of-view" (NFOV) images. These images capture only a small portion of the visual scene. In contrast, 360° images, increasingly common with VR/AR technologies, capture the entire environment. This expansive view introduces significant challenges for VQA:

  • Spatial Understanding: Machines need to reason about objects and their relationships across a much larger and more complex spatial layout.
  • Data Scarcity: There's a lack of large-scale VQA datasets specifically for 360° imagery.
  • Distortion: 360° images, often stored in an equirectangular format, suffer from significant spatial distortion, making it difficult for standard computer vision models to process them effectively.

This paper aims to bridge this gap by introducing a new VQA task and dataset for 360° images and developing models capable of handling these unique challenges.

Methods

The authors tackle the challenges of VQA 360° with a two-pronged approach:

  1. Cubemap Representation: To mitigate the spatial distortion of equirectangular images, they represent each 360° image as six non-overlapping cubemaps. Each cubemap captures a specific spatial region with less distortion, allowing for more localized and accurate feature extraction. This naturally leads to a bottom-up processing architecture, starting with local understanding and moving to global reasoning.
  2. Cubemap-based VQA Model: They propose a novel architecture that leverages the cubemap representation.
  3. Feature Extraction: Standard VQA models (like MLB) are applied to each individual cubemap to extract local multimodal features.
  4. Multi-level Attention: An attention mechanism is used to aggregate information from these cubemaps. This attention is guided by the question and considers the location of each cubemap.
  5. Attention Diffusion: To further enhance spatial reasoning, they introduce an "attention diffusion" mechanism. This allows the model to learn how to shift attention between cubemaps based on the question, enabling it to reason about relative spatial positions across the entire 360° scene.

They also experiment with an "equirectangular-based" approach, where standard VQA models are applied directly to the distorted 360° images, serving as a baseline.

Datasets

  • VQA 360° Dataset (New): The authors introduce the first real-world VQA 360° dataset, containing approximately 17,000 image-question-answer triplets.
  • Image Source: Collected from indoor scenes in the Stanford 2D-3D and Matterport3D datasets.
  • Question Types: Questions are categorized into five types: "scene," "exist," "counting," "property," and "spatial." The "spatial" type is particularly emphasized to encourage reasoning about relative positions.
  • Annotation: Human annotators provided answers, with efforts made to minimize dataset bias and ensure answer uniformity across question types.

Key Insights and Ideas

  • 360° Images Demand New Approaches: Standard VQA models struggle with the inherent distortion and vast spatial coverage of 360° images, highlighting the need for specialized architectures.
  • Cubemaps are Crucial for Spatial Reasoning: Representing 360° images as cubemaps significantly reduces distortion and provides a more structured input for spatial reasoning tasks.
  • Multi-level Attention is Key: Simply averaging cubemap features is insufficient. A sophisticated attention mechanism that considers question context and spatial relationships between cubemaps is vital for accurate VQA on 360° images.
  • Human-Machine Gap Persists: Despite advancements, a significant performance gap remains between human and machine performance on VQA 360°, especially for "counting," "property," and "spatial" questions, indicating areas for future research.

Key Results

  • Cubemap-based Model Superiority: The proposed cubemap-based model with "TUCKER&DIFFUSION" consistently outperformed equirectangular-based models across various metrics, demonstrating the effectiveness of their approach.
  • Top Performance: The best cubemap-based model achieved an overall average accuracy of 58.66% on the VQA 360° test set, significantly better than the best equirectangular model (54.77%).
  • Spatial Question Improvement: The "TUCKER&DIFFUSION" model showed a notable improvement on "spatial" question types, highlighting the benefit of attention diffusion.
  • Human Performance: Human annotators achieved an accuracy of 84.05% on a subset of the dataset, indicating the dataset's quality and the remaining challenge for machines.

Limitations and Future Work

  • Cubemap Splitting: Objects can be split across cubemap boundaries, potentially hindering detection and reasoning.
  • Hand-crafted Cubemap Cropping: The current cubemap projection is fixed, and more adaptive methods are needed.
  • Future Directions:
    • Image-dependent Cropping: Develop methods that detect object regions and adapt cropping accordingly.
  • Back-projection Mechanisms: Project detected objects back into the 360° environment for more robust reasoning.
  • Increased Question Diversity: Incorporate more human effort to generate a wider variety of questions.
  • Advanced Object Detection: Improve object detection specifically for 360° images to better handle small or split objects.

Summary updated

Abstract

In this work, we introduce VQA 360, a novel task of visual question answering on 360 images. Unlike a normal field-of-view image, a 360 image captures the entire visual content around the optical center of a camera, demanding more sophisticated spatial understanding and reasoning. To address this problem, we collect the first VQA 360 dataset, containing around 17,000 real-world image-question-answer triplets for a variety of question types. We then study two different VQA models on VQA 360, including one conventional model that takes an equirectangular image (with intrinsic distortion) as input and one dedicated model that first projects a 360 image onto cubemaps and subsequently aggregates the information from multiple spatial resolutions. We demonstrate that the cubemap-based model with multi-level fusion and attention diffusion performs favorably against other variants and the equirectangular-based models. Nevertheless, the gap between the humans' and machines' performance reveals the need for more advanced VQA 360 algorithms. We, therefore, expect our dataset and studies to serve as the benchmark for future development in this challenging task. Dataset, code, and pre-trained models are available online.
BibTeX
@inproceedings{chou2020visual,
  author = {Shih-Han Chou and Wei-Lun Chao and Wei-Sheng Lai and Min Sun and Ming-Hsuan Yang},
  title = {{Visual Question Answering on 360° Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093452},
}
Figure