Abstract: Embodied Question Answering (EQA) is a relatively new task where an agent is asked to answer questions about its environment from egocentric perception. EQA makes the fundamental assumption that every question, e.g., "what color is the car?", has exactly one target ("car") being inquired about. This assumption puts a direct limitation on the abilities of the agent. We present a generalization of EQA - Multi-Target EQA (MT-EQA). Specifically, we study questions that have multiple targets in them, such as "Is the dresser in the bedroom bigger than the oven in the kitchen?", where the agent has to navigate to multiple locations ("dresser in bedroom", "oven in kitchen") and perform comparative reasoning ("dresser" bigger than "oven") before it can answer a question. Such questions require the development of entirely new modules or components in the agent. To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module. The program generator converts the given question into sequential executable sub-programs; the navigator guides the agent to multiple locations pertinent to the navigation-related sub-programs; and the controller learns to select relevant observations along its path. These observations are then fed to the VQA module to predict the answer. We perform detailed analysis for each of the model components and show that our joint model can outperform previous methods and strong baselines by a significant margin.
Introduction
Embodied Question Answering (EQA) represents a frontier in AI where agents must bridge the gap between natural language understanding, active vision, and physical navigation. Traditional EQA benchmarks, such as EQA-v1, typically focus on single-target queries like "What color is the car?" While these tasks test an agent's ability to locate an object, they do not capture the complexity of human intelligence, which often involves multi-step reasoning and comparison across different spatial contexts.
Multi-Target EQA (MT-EQA) addresses this limitation by requiring agents to navigate to multiple locations and reason about several objects sequentially. For instance, answering "Is the dressing table in the bedroom the same color as the sink in the bathroom?" requires an agent to visit two distinct rooms, identify specific objects, and maintain a memory of visual attributes to perform a final comparison.
A Modular Architecture for Sequential Reasoning
To handle the increased complexity of MT-EQA, this research moves away from monolithic end-to-end models in favor of a modular architecture. This approach allows for the decoupling of high-level task planning from low-level navigation and visual reasoning.
1. Program Generator
The process begins with the Program Generator, which decomposes a natural language question into a sequence of functional sub-programs. This allows the agent to treat a complex query as a series of manageable tasks.
2. Navigator and Controller
The Navigator is an LSTM-based module that predicts movement actions based on egocentric visual frames. Crucially, the system employs separate navigation policies for finding general rooms versus specific objects. The Controller acts as the central executive, managing the execution of sub-programs and deciding when a target has been reached (the SELECT action). This modularity prevents the "forgetting" problem common in simpler architectures when transitioning between targets.
3. VQA Module
Once the agent has successfully navigated to all required targets, the VQA Module processes the stored visual features. It applies the relevant comparison operator (e.g., "bigger than" or "same color") to predict the final answer.
Training and Dataset
The authors introduced the MT-EQA Dataset, comprising 19,287 questions across 588 unique environments in the House3D simulation platform. The dataset includes six categories of comparative reasoning involving color, size, and distance. To ensure the agent actually explores the environment rather than exploiting linguistic biases, the researchers utilized "entropy filtering" to remove questions that could be guessed from text alone.
The training strategy involves two stages:
- Imitation Learning (IL): The model is first trained on shortest-path data to establish a baseline for navigation.
- Reinforcement Learning (RL): The Navigator is fine-tuned using RL with a reward function based on distance to the target and a "Best-View" metric. This metric rewards the agent for achieving a high Intersection-over-Union (IoU) between the target object and the center of its field of view, ensuring the VQA module receives high-quality visual input.
Key Results
The modular joint model significantly outperformed previous methods and strong baselines:
- Accuracy: The full model with RL and the modular Controller achieved an overall accuracy of 61.45%, compared to 53.24% for a non-controller baseline.
- Navigation Success: RL fine-tuning was critical, improving the "hit accuracy" (the ability to correctly identify the target object at the end of a sequence) from 20% to 33%.
- Embodiment Validation: Language-only baselines performed near chance (~50%), confirming that the MT-EQA dataset successfully requires visual perception and navigation to solve.
Conclusion
MT-EQA represents a significant advancement in embodied AI by introducing tasks that require sequential navigation and comparative reasoning. By utilizing a modular architecture and RL-based fine-tuning with view-quality rewards, agents can overcome the limitations of single-target EQA. Future work will look toward more flexible natural language translation and operating in dynamic, non-static environments.
Summary updated