Visual Question Answering on 360deg Images

Shih-Han Chou, Wei-Lun Chao, Wei-Sheng Lai, Min Sun, Ming-Hsuan YangWACV 2020

TL;DR

This paper introduces VQA 360°, a novel task and dataset for visual question answering on 360° images, addressing the challenges of spatial distortion and comprehensive scene understanding. They propose a cubemap-based model with multi-level attention and diffusion mechanisms that significantly outperforms equirectangular-based approaches, though a notable gap remains between machine and human performance.

What Problem Does This Paper Address?

Visual Question Answering (VQA) has seen tremendous progress, but existing work primarily focuses on normal field-of-view (NFOV) images. This limits machines' ability to understand and reason about the full visual context of a scene, which is crucial for applications like robotics and immersive technologies. 360° images, while offering a complete spherical view, introduce significant spatial distortion in standard representations (equirectangular projection) and require more sophisticated spatial reasoning to answer questions accurately. The authors identify a lack of suitable datasets and models for this challenging domain.

Methods

The paper proposes two main approaches:

  • Equirectangular-based Models: These models adapt existing VQA architectures (like MLB) to process the 360° image in its equirectangular format. Strategies like cropping, resizing, or using average pooling are explored, but these methods struggle with the inherent spatial distortion.
  • Cubemap-based Models: This is the core novel contribution. The 360° image is first converted into six cubemaps, each representing a face of a cube. This representation significantly reduces spatial distortion. A bottom-up architecture is then employed:
  • Feature Extraction: Standard VQA models are applied to each cubemap individually.
  • Multi-level Attention: An attention mechanism is used to aggregate features from the different cubemaps, considering their spatial locations and the question.
  • Attention Diffusion: A further enhancement learns a "diffusion matrix" to allow attention to shift between cubemaps, enabling more nuanced spatial reasoning (e.g., looking from a window to its right). This is achieved through Tucker fusion for weight computation and a diffusion mechanism.

Datasets

  • VQA 360° Dataset: The authors introduce the first real-world VQA 360° dataset, containing approximately 17,000 image-question-answer triplets.
  • Images: Collected from the Stanford 2D-3D and Matterport3D datasets, focusing on indoor scenes.
  • Questions: Generated using templates covering five types: "scene," "exist," "counting," "property," and "spatial." The questions are designed to require spatial understanding and reasoning.
  • Answers: Human-annotated, with efforts to minimize dataset bias.

Key Insights and Ideas

  • Spatial Distortion is a Major Hurdle: Equirectangular representations of 360° images inherently distort spatial relationships, significantly impacting VQA performance.
  • Cubemaps as a Solution: Decomposing a 360° image into cubemaps effectively mitigates spatial distortion, providing a more manageable input for VQA models.
  • Multi-level Attention is Crucial: Simply pooling features from cubemaps is insufficient. A sophisticated attention mechanism that considers cubemap locations and question context is vital for effective reasoning.
  • Attention Diffusion Enhances Spatial Reasoning: The ability to "diffuse" attention across cubemaps, guided by the question, further boosts performance, particularly for spatial questions.
  • Pre-training is Beneficial: Leveraging models pre-trained on large NFOV VQA datasets significantly improves performance, even when fine-tuned on the VQA 360° dataset.

Key Results

  • The Cubemap-based model with TUCKER&DIFFUSION achieved the best performance, reaching 58.66% overall accuracy on the VQA 360° test set.
  • This cubemap-based approach outperformed all equirectangular-based models by a margin of 3-4%, highlighting the importance of handling spatial distortion.
  • The TUCKER&DIFFUSION model showed a notable improvement over the TUCKER model (without diffusion), especially on "spatial" question types, demonstrating the value of the diffusion mechanism.
  • Human evaluation showed 84.05% overall accuracy, indicating a significant gap of approximately 25% between human and machine performance, particularly in "counting," "property," and "spatial" question types.
  • The dataset exhibits low language bias, requiring machines to rely on visual information for answers.

Limitations and Future Work

  • Cubemap Cropping Artifacts: The current cubemap generation method can split objects across cubemap boundaries, potentially hindering performance.
  • Need for More Sophisticated Reasoning: The remaining gap between human and machine performance suggests the need for more advanced reasoning mechanisms, especially for complex spatial relationships.
  • Future Directions:
    • Developing image-dependent cropping that better handles object boundaries.
  • Implementing back-projection mechanisms to reason about objects in the 360° environment.
  • Increasing question diversity beyond template-based generation, potentially through more human annotation effort.
  • Exploring more diverse VQA backbones and attention mechanisms.

Summary updated

BibTeX
@inproceedings{chou2020visual,
  author = {Shih-Han Chou and Wei-Lun Chao and Wei-Sheng Lai and Min Sun and Ming-Hsuan Yang},
  title = {{Visual Question Answering on 360deg Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure