Walking with MIND: Mental Imagery eNhanceD Embodied QA

Juncheng Li, Siliang Tang, Fei Wu, Yueting Zhuang

TL;DR

Researchers have developed the MIND (Mental Imagery eNhanceD) module, which allows Embodied QA agents to "imagine" the outcomes of their actions before taking them. By modeling environment dynamics and using predicted mental images as subgoals, the agent achieves superior navigation efficiency, better generalization to new rooms, and provides a window into its "thought process" for human interpretability.

What Problem Does This Paper Address?

The task is Embodied Question Answering (EQA): an AI agent is dropped into a random location in a simulated house and asked a question like, "What color is the refrigerator?" To answer, the agent must navigate through rooms, find the object, and process the visual data.

Standard approaches often fail because they try to map raw pixels directly to primitive actions (move forward, turn left) without a high-level understanding of the world. These agents lack a "mental model"—they don't realize that to see a refrigerator, they might first need to "imagine" finding a door to leave the current room. This leads to "blind" navigation, where agents bump into walls or fail to generalize when placed in a slightly different house layout.

Methods

The authors introduce the MIND module, which consists of two main parts designed to mimic human mental imagery:

  1. Mental Autoencoder ($\beta$-VAE): This component learns to compress complex RGB images into "disentangled" latent representations. Instead of just seeing a grid of pixels, the agent learns abstract factors (like room direction or scale). This makes the agent's internal world-model more robust and easier to transfer to new environments.
  2. Imagery Model (LSTM + MDN): This is the "engine" of imagination. It uses a Long Short-Term Memory (LSTM) network combined with a Mixture Density Network (MDN) to predict what the next mental state will look like given a specific action.

The "Walking with MIND" Framework

The agent uses a hierarchical structure (Planner and Controller). The Planner doesn't just look at the current frame; it looks at its "imagined" future state.

To train this, the authors introduced a novel Planned Reward ($r_m$). Unlike standard rewards that only trigger when the agent gets physically closer to a goal, the planned reward gives the agent "points" if its imagined subgoal actually increases the probability of answering the question correctly. This encourages the agent to form task-related mental images (e.g., imagining the end of a hallway) rather than just random snapshots.

Datasets

The model was evaluated on the EQA Dataset, which is built on the House3D environment.

  • Scale: ~9,000 questions across 774 environments.
  • Diversity: Questions cover object locations, colors, and spatial prepositions (e.g., "What is next to the coffee machine in the kitchen?").
  • Split: The test set uses entirely novel environments that the agent never saw during training to strictly test generalization.

Key Insights and Ideas

  • Subgoals as Images: The core "aha moment" is treating predicted mental images as short-term subgoals. By "seeing" the door in its mind's eye before moving, the agent avoids the trial-and-error behavior seen in previous models.
  • Disentanglement Matters: Using a $\beta$-VAE to learn latent factors allows the agent to generalize better. If the agent understands the "concept" of a corridor or a doorway independently of specific textures, it can navigate a new house much more effectively.
  • Interpretability: Because the MIND module generates actual images (via the decoder), researchers can visualize what the agent is "thinking." If the agent is heading toward a bedroom, its mental imagery shows a bedroom, making its behavior explainable to humans.

Key Results

The MIND agent was compared against several baselines, including PACMAN and Neural Modular Control (NMC):

  • Navigation Efficiency ($d_\Delta$): The MIND agent consistently outperformed others in reaching the target. At a starting distance of 50 actions away ($T_{-50}$), MIND achieved a distance change ($d_\Delta$) of 1.65, compared to PACMAN's 1.51.
  • Answering Accuracy: MIND achieved 54.83% accuracy at $T_{-10}$, significantly higher than the standard PACMAN (48.48%).
  • Sample Efficiency: Perhaps most impressively, the MIND agent converged much faster. Even when trained on only 200 examples (roughly 1/35th of the full training set), the MIND agent maintained functional navigation, whereas PACMAN "almost does not work" with so little data.

Limitations and Future Work

  • The QA Bottleneck: The authors acknowledge that while navigation improved significantly, the "answering" part of EQA remains a bottleneck. Even if the agent finds the object, it sometimes fails to answer correctly due to the complexity of visual reasoning.
  • Static Environments: The current model operates in static, simulated environments. Moving to dynamic environments (where objects might move or change) would require a much more complex imagery model.
  • Action Granularity: The agent still relies on a fixed set of primitive actions. Future work could involve the MIND module generating more continuous or complex "imagined" trajectories.

Summary updated

Abstract

The EmbodiedQA is a task of training an embodied agent by intelligently navigating in a simulated environment and gathering visual information to answer questions. Existing approaches fail to explicitly model the mental imagery function of the agent, while the mental imagery is crucial to embodied cognition, and has a close relation to many high-level meta-skills such as generalization and interpretation. In this paper, we propose a novel Mental Imagery eNhanceD (MIND) module for the embodied agent, as well as a relevant deep reinforcement framework for training. The MIND module can not only model the dynamics of the environment (e.g. 'what might happen if the agent passes through a door') but also help the agent to create a better understanding of the environment (e.g. 'The refrigerator is usually in the kitchen'). Such knowledge makes the agent a faster and better learner in locating a feasible policy with only a few trails. Furthermore, the MIND module can generate mental images that are treated as short-term subgoals by our proposed deep reinforcement framework. These mental images facilitate policy learning since short-term subgoals are easy to achieve and reusable. This yields better planning efficiency than other algorithms that learn a policy directly from primitive actions. Finally, the mental images visualize the agent's intentions in a way that human can understand, and this endows our agent's actions with more interpretability. The experimental results and further analysis prove that the agent with the MIND module is superior to its counterparts not only in EQA performance but in many other aspects such as route planning, behavioral interpretation, and the ability to generalize from a few examples.
BibTeX
@inproceedings{li2019walking,
  author = {Juncheng Li and Siliang Tang and Fei Wu and Yueting Zhuang},
  title = {{Walking with MIND: Mental Imagery eNhanceD Embodied QA}},
  booktitle = {ACM Multimedia},
  year = {2019},
  doi = {10.1145/3343031.3351017},
}

Loading related content...

Figure