Walking with MIND: Mental Imagery eNhanceD Embodied QA

Juncheng Li, Siliang Tang, Fei Wu, Yueting Zhuang

Abstract

The Embodied Question Answering (EQA) task involves training an agent to navigate a simulated environment and gather visual information to answer natural language questions. Existing approaches often fail to explicitly model the mental imagery function, which is crucial for embodied cognition and high-level meta-skills like generalization. We propose a novel Mental Imagery eNhanceD (MIND) module and a deep reinforcement learning framework. The MIND module models environment dynamics and helps the agent understand spatial relationships (e.g., "refrigerators are usually in the kitchen"). This knowledge enables faster learning and better policy location. Furthermore, the MIND module generates mental images as short-term subgoals, improving planning efficiency over primitive action-based methods. These mental images also enhance interpretability by visualizing the agent's intentions. Experimental results demonstrate that the MIND agent excels in navigation, route planning, behavioral interpretation, and few-shot generalization.

Introduction

Embodied Question Answering (EQA) represents a significant challenge in AI, requiring an agent to be dropped into a novel environment and asked a question such as, "What color is the refrigerator?" To succeed, the agent must navigate through rooms, locate the target object, and process visual data to formulate an answer.

Overview of the Embodied QA task: an agent navigates a simulated environment to answer a question about an object's color.

Standard EQA approaches typically map raw pixels directly to primitive actions (e.g., move forward, turn left). However, these methods often lack a high-level "mental model" of the world, leading to inefficient navigation and poor generalization to new layouts. Without the ability to "imagine" the outcome of actions, agents may struggle to plan complex routes, such as navigating through a door to reach a different room.

The MIND Module and Framework

To address these limitations, we introduce the Mental Imagery eNhanceD (MIND) module. This module is designed to mimic human mental imagery, allowing the agent to predict future states and use them as subgoals.

Architecture

The MIND framework utilizes a hierarchical structure consisting of a Planner (PLNR) and a Controller (CTRL). The core of the system is the MIND module, which integrates two key components:

  1. Mental Autoencoder (beta-VAE): This component learns to compress complex RGB images into disentangled latent representations (m_t). By learning abstract factors like room direction or scale, the agent develops a robust internal world model that generalizes well to unseen environments.
  2. Imagery Model (LSTM + MDN): This "imagination engine" uses a Long Short-Term Memory (LSTM) network combined with a Mixture Density Network (MDN) to predict the next mental state (m_{t+1}) given the current state and a proposed action (a_t).
The MIND architecture: A hierarchical framework where the Imagery Model generates mental subgoals ($M_t$) to guide the Planner and Controller.

Planned Reward

A critical innovation is the Planned Reward (r_m). Unlike traditional rewards that only trigger upon physical proximity to a goal, r_m rewards the agent if its imagined future state increases the probability of answering the question correctly. This encourages the agent to form task-relevant mental images, such as the end of a hallway or a specific doorway, rather than arbitrary snapshots.

Experimental Evaluation

We evaluated the MIND agent using the EQA Dataset based on the House3D environment. The dataset includes approximately 9,000 questions across 774 diverse environments, covering object locations, colors, and spatial prepositions. To rigorously test generalization, the test set consists entirely of novel house layouts not seen during training.

The MIND agent demonstrated superior performance across several metrics:

  • Navigation Efficiency (d_delta): When starting 50 actions away from the target (T_{-50}), the MIND agent achieved a distance change of 1.65, significantly outperforming the PACMAN baseline (1.51).
  • Answering Accuracy: At T_{-10}, MIND achieved 54.83% accuracy, compared to 48.48% for PACMAN.

Sample Efficiency and Interpretability

The MIND agent is a remarkably fast learner. Even when trained on only 200 examples—roughly 1/35th of the full training set—it maintained functional navigation capabilities, whereas baselines failed to converge. Furthermore, the MIND module's ability to generate actual images from latent states allows researchers to visualize the agent's "thoughts," providing a window into its decision-making process and enhancing the interpretability of its actions.

Conclusion

The MIND module significantly improves the performance and interpretability of embodied agents by providing them with a "mind's eye." By treating predicted mental images as subgoals, the agent achieves higher navigation efficiency and better generalization. Future work will focus on overcoming the visual reasoning bottleneck in the answering phase and extending the model to dynamic environments.

Summary updated

Abstract

The EmbodiedQA is a task of training an embodied agent by intelligently navigating in a simulated environment and gathering visual information to answer questions. Existing approaches fail to explicitly model the mental imagery function of the agent, while the mental imagery is crucial to embodied cognition, and has a close relation to many high-level meta-skills such as generalization and interpretation. In this paper, we propose a novel Mental Imagery eNhanceD (MIND) module for the embodied agent, as well as a relevant deep reinforcement framework for training. The MIND module can not only model the dynamics of the environment (e.g. 'what might happen if the agent passes through a door') but also help the agent to create a better understanding of the environment (e.g. 'The refrigerator is usually in the kitchen'). Such knowledge makes the agent a faster and better learner in locating a feasible policy with only a few trails. Furthermore, the MIND module can generate mental images that are treated as short-term subgoals by our proposed deep reinforcement framework. These mental images facilitate policy learning since short-term subgoals are easy to achieve and reusable. This yields better planning efficiency than other algorithms that learn a policy directly from primitive actions. Finally, the mental images visualize the agent's intentions in a way that human can understand, and this endows our agent's actions with more interpretability. The experimental results and further analysis prove that the agent with the MIND module is superior to its counterparts not only in EQA performance but in many other aspects such as route planning, behavioral interpretation, and the ability to generalize from a few examples.
BibTeX
@inproceedings{li2019walking,
  author = {Juncheng Li and Siliang Tang and Fei Wu and Yueting Zhuang},
  title = {{Walking with MIND: Mental Imagery eNhanceD Embodied QA}},
  booktitle = {ACM Multimedia},
  year = {2019},
  doi = {10.1145/3343031.3351017},
}
Figure