VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering.

Catalina Cangea, Eugene Belilovsky, Aaron C. CourvilleBMVC 2019 · OpenReview

TL;DR

This paper introduces VideoNavQA, a new benchmark dataset and task that decouples the navigation aspect from Embodied Question Answering (EQA) to focus on complex visual reasoning. They evaluate various VQA models on this dataset, finding that while current methods show promise, there's still a significant gap to human-level performance, highlighting the need for more advanced visual reasoning techniques.

What Problem Does This Paper Address?

Embodied Question Answering (EQA) aims to create agents that can navigate 3D environments and answer questions based on their visual input. However, existing EQA approaches often combine complex navigation and reasoning, making it difficult to isolate and improve the reasoning component. Initial attempts using standard vision-language models and reinforcement learning have struggled with the inherent complexity of EQA. This paper addresses this by proposing a new task, VideoNavQA, which simplifies the navigation aspect by providing near-optimal trajectories. This allows researchers to focus on the visual reasoning challenges, specifically by increasing the complexity and variety of questions compared to previous EQA datasets. The goal is to understand how well current Visual Question Answering (VQA) techniques can perform in this more challenging reasoning-focused EQA paradigm.

Methods

The authors introduce the VideoNavQA dataset, which consists of question-video pairs generated within the House3D environment. The key innovation is providing near-optimal navigation trajectories, meaning the agent is already in a position to answer the question based on the visual information in the video. They then adapt and evaluate several VQA models on this dataset. These include:

  • Language-only baselines: LSTM and Bag-of-Words (BoW) models to establish a lower bound based on language understanding alone.
  • Vision-only baselines: Per-frame processing VGG-style CNN with an LSTM classifier (V-CNN2D) and a C3D-like CNN (V-CNN3D) to assess performance based solely on visual input.
  • Concatenation models: Combining question and video representations by concatenating them and feeding them into an MLP.
  • FiLM-based models: Feature-wise Linear Modulation (FiLM) is extended to handle the temporal dimension of videos. This involves processing each frame with ResBlocks and then conditioning these features using the question embedding. Two aggregation mechanisms are explored: attention (FiLM AT) and global max-pooling (FiLM GP).
  • Temporal Multi-hop model: An extension of FiLM that conditions video features by attending over question encoder hidden states, incorporating temporal context from previous frames.
  • MAC (Compositional Attention Networks): Adapted for video by applying a MAC model after a 2D-CNN and integrating it over time with an LSTM.

Datasets

The primary dataset introduced in this paper is VideoNavQA. It is generated using the House3D virtual environment and contains approximately 101,000 question-video pairs. The dataset features 28 types of questions across 8 categories, offering a more complex reasoning challenge than previous EQA datasets. The paper also references existing datasets like EQAv1 and CLEVR for comparison and context.

Key Insights and Ideas

  • Decoupling Navigation for Focused Reasoning: By providing near-optimal trajectories, the paper effectively isolates the visual reasoning challenge in EQA, allowing for a more targeted evaluation of VQA models in this context.
  • Increased Question Complexity: VideoNavQA's 28 question types significantly expand the reasoning capabilities tested compared to prior EQA datasets, pushing the boundaries of what current models can handle.
  • VQA Models Show Promise but Gap Remains: While adapted VQA models perform significantly better than language-only baselines, their performance is still far from human-level, indicating that current VQA architectures may not fully capture the nuances of complex visual reasoning in video.
  • Importance of Temporal Reasoning: Models that explicitly incorporate temporal information (like FiLM variants and the Temporal Multi-hop model) show competitive performance, suggesting that understanding the sequence of visual information is crucial.

Key Results

The paper reports the following key results on the VideoNavQA test set:

  • Overall Accuracy: Concat-CNN2D achieved the highest accuracy at 64.47%. FiLM GP and FiLM AT performed similarly, around 63.79% and 64.08% respectively. The Temporal Multi-hop model achieved 63.53%, outperforming language-only models by a significant margin.
  • Language-only baselines: LSTM achieved 56.49% accuracy, significantly better than BoW at 49.02%.
  • Question Category Performance: All models performed best on "existence" questions. "Identifying object types" was the hardest task. MAC excelled at identifying room locations and counting, while Concat-CNN2D was strong in comparing counts.
  • Comparison to Baselines: All evaluated models significantly improved upon the language-only baselines, demonstrating the utility of visual information.

Limitations and Future Work

  • Navigation is Trivialized: The paper acknowledges that by providing near-optimal trajectories, the navigation aspect of EQA is simplified. Future work could explore integrating more complex navigation strategies.
  • Gap to Human Performance: Despite improvements, the gap between model performance and human-level reasoning remains substantial. The authors suggest that more investigation into novel VQA techniques is needed to better isolate and utilize visual information from videos.
  • Need for More Advanced Models: The results suggest that current VQA models, even when adapted for video, may not be sufficient for the complex reasoning required in VideoNavQA. Future research should focus on developing architectures that can better handle rich temporal and contextual information.

Summary updated

Abstract

Embodied Question Answering (EQA) is a recently proposed task, where an agent is placed in a rich 3D environment and must act based solely on its egocentric input to answer a given question. The desired outcome is that the agent learns to combine capabilities such as scene understanding, navigation and language understanding in order to perform complex reasoning in the visual world. However, initial advancements combining standard vision and language methods with imitation and reinforcement learning algorithms have shown EQA might be too complex and challenging for these techniques. In order to investigate the feasibility of EQA-type tasks, we build the VideoNavQA dataset that contains pairs of questions and videos generated in the House3D environment. The goal of this dataset is to assess question-answering performance from nearly-ideal navigation paths, while considering a much more complete variety of questions than current instantiations of the EQA task. We investigate several models, adapted from popular VQA methods, on this new benchmark. This establishes an initial understanding of how well VQA-style methods can perform within this novel EQA paradigm.
BibTeX
@inproceedings{cangea2019videonavqa,
  author = {Catalina Cangea and Eugene Belilovsky and Aaron C. Courville},
  title = {{VideoNavQA: Bridging the Gap between Visual and Embodied Question Answering.}},
  booktitle = {BMVC},
  year = {2019},
  url = {https://openreview.net/forum?id=1rpi0cfwSN},
}
Figure