Abstract
Embodied Question Answering (EQA) is a relatively new task where an agent is asked to answer questions about its environment from egocentric perception. EQA as introduced in [8] makes the fundamental assumption that every question, e.g., "what color is the car?", has exactly one target ("car") being inquired about. This assumption puts a direct limitation on the abilities of the agent. We present a generalization of EQA -- Multi-Target EQA (MT-EQA). Specifically, we study questions that have multiple targets in them, such as "Is the dresser in the bedroom bigger than the oven in the kitchen?", where the agent has to navigate to multiple locations ("dresser in bedroom", "oven in kitchen") and perform comparative reasoning ("dresser" bigger than "oven") before it can answer a question. Such questions require the development of entirely new modules or components in the agent. To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module. The program generator converts the given question into sequential executable sub-programs; the navigator guides the agent to multiple locations pertinent to the navigation-related sub-programs; and the controller learns to select relevant observations along its path. These observations are then fed to the VQA module to predict the answer. We perform detailed analysis for each of the model components and show that our joint model can outperform previous methods and strong baselines by a significant margin.
Introduction
The field of Embodied AI has seen significant progress with the introduction of Embodied Question Answering (EQA). In the standard EQA paradigm, an agent is placed in a 3D environment and must navigate to find a single target object to answer a query, such as "What color is the car?". However, this single-target assumption is overly restrictive for real-world applications where complex reasoning often involves multiple entities across different locations.
We introduce Multi-Target EQA (MT-EQA), which requires agents to handle multi-part queries involving comparative reasoning (e.g., "Is the dressing table in the bedroom the same color as the sink in the bathroom?"). This task necessitates multi-stop navigation, temporal memory to store features of the first target, and a compositional reasoning module to perform the final comparison.
The MT-EQA Framework
To address the complexities of MT-EQA, we propose a modular architecture that decouples navigation from visual reasoning. The system consists of four primary components:
- Program Generator: This module parses the natural language question into a sequence of executable sub-programs. For instance, a comparison question is broken down into navigation steps followed by a query step.
- Navigator: A dual-module system where one LSTM-based module handles room-level navigation and another handles object-level navigation within a room.
- Controller: Acting as the system's manager, the controller executes sub-programs sequentially. It predicts a "SELECT" action to capture optimal visual frames when a target is identified.
- VQA Module (cVQA): A compositional reasoning module that processes the "frozen" features selected by the controller to produce the final answer.
Dataset and Training
We developed the MT-EQA Dataset using the House3D simulation platform and SUNCG indoor scenes. The dataset contains 19,287 questions across 588 unique environments, covering 61 object types and 8 room types. Questions are categorized into six types, including comparisons of color, size, and distance.
Training is conducted in two stages. First, Imitation Learning (IL) teaches the agent to follow shortest-path trajectories. Second, Reinforcement Learning (RL) fine-tunes the navigator to optimize for visual clarity of the target, improving the quality of features provided to the VQA module.
Results and Discussion
Our full modular model (Nav(RL)+Ctrl+cVQA) achieved an accuracy of 61.45%, significantly outperforming standard EQA baselines. Key findings include:
- Navigation Efficiency: RL fine-tuning reduced average episode length from 153 to 127 steps and increased target "hit accuracy" from 19% to 33%.
- Visual Quality: The "SELECT" mechanism allows the agent to identify frames with higher Intersection-over-Union (IOU) relative to the target, which is critical for accurate reasoning.
- Scaling Challenges: While the model performs well on "easy" questions (~72% accuracy), performance drops on "hard" questions involving long-range navigation (~55%), indicating that long-term spatial reasoning remains a significant challenge.
Conclusion
MT-EQA represents a significant advancement in Embodied AI by moving beyond single-object queries to complex, multi-target reasoning. Our modular approach demonstrates that decoupling navigation from reasoning and employing a selective attention mechanism for visual features are effective strategies for handling complex environmental queries. Future work will focus on handling open-ended questions and improving navigation in more dynamic and complex environments.
Summary updated