ForeSI: Success-Aware Visual Navigation Agent

Mohammad Mahdi Kazemi Moghaddam, Ehsan Abbasnejad, Qi Wu, Qinfeng (Javen) Shi, Anton van den Hengel

Introduction

Object-goal visual navigation requires an agent to locate a specific object, such as a "toaster," within a previously unseen 3D environment. Most contemporary approaches rely on model-free Reinforcement Learning (RL). While effective for reactive tasks, these agents often lack long-term foresight and suffer from a "forgetting" problem during extended trajectories, where obstacle avoidance or complex layouts distract them from the ultimate goal.

Conceptual overview of ForeSI. The agent imagines a critical sub-goal to navigate toward a target object (toaster) in an unseen environment.

To address these challenges, we introduce the Foresight Sub-goal Imaginator (ForeSI). ForeSI bridges the gap between model-free and model-based RL by augmenting agents with the ability to "imagine" a successful future state—a critical sub-goal—at the start of their journey.

Methodology

ForeSI does not attempt to predict every future frame, which is computationally prohibitive in complex 3D scenes. Instead, it focuses on a single, high-level latent representation of a state critical for success.

The framework consists of several key innovations:

  1. Hindsight Sub-goal Identification: Using an attention mechanism, the agent reviews successful training episodes to identify states most correlated with high rewards. These "turning points" are labeled as sub-goals.
  2. The Imaginator ($f_w$): This module is trained to predict the latent representation of the identified sub-goal at the beginning of an episode, conditioned on the initial state and the target object.
  3. Success-Aware Noise: To handle early-stage inaccuracies in the imaginator, Gaussian noise is injected into the imagined state. This noise decays as the agent's success rate improves, ensuring stable training.
  4. Policy Conditioning: The agent's policy is conditioned on both current visual observations and the imagined sub-goal, providing a persistent "beacon" for navigation.

Experimental Setup

The ForeSI framework was evaluated using the AI2-THOR simulator across various indoor environments, including kitchens, bedrooms, bathrooms, and living rooms. The training involved 20 scenes per room type, with 5 held-out scenes for testing. ForeSI was integrated into multiple baselines, including A3C, A3C+MAML (meta-learning), and A3C+ORG (object relationship graphs), to demonstrate its modularity and robustness across different architectures.

Results and Discussion

The addition of ForeSI yielded consistent improvements across all primary metrics. The Success Rate (SR) saw an absolute improvement of approximately 5% over standard A3C and 3% when integrated with advanced object-detector baselines. Efficiency, measured by Success weighted by Path Length (SPL), improved by roughly 2%, indicating more direct navigation paths.

Qualitative comparison of navigation trajectories. Agents using ForeSI (top) exhibit more direct paths to targets compared to reactive baselines (bottom).

The performance gains were most significant in large, complex environments like kitchens and living rooms, where trajectories are longer. In these settings, the imagined sub-goal effectively mitigated the "forgetting" problem, allowing the agent to maintain focus on the target even when it was occluded or far away. Qualitative analysis shows that while baseline agents often wander or circle in place, ForeSI-equipped agents move decisively toward the target's likely location.

Conclusion

ForeSI demonstrates that predicting a latent "imagined" future state can significantly enhance the performance of model-free navigation agents. By providing a goal-oriented beacon, the system improves both success rates and path efficiency in unseen 3D environments. Future work will explore learning from failed trajectories and dynamically predicting sequences of sub-goals for even more complex navigation tasks.

Summary updated

Abstract

In this work, we present a method to improve the efficiency and robustness of the previous model-free Reinforcement Learning (RL) algorithms for the task of object-goal visual navigation. Despite achieving state-of-the-art results, one of the major drawbacks of those approaches is the lack of a forward model that informs the agent about the potential consequences of its actions, i.e., being model-free. In this work, we augment the model-free RL with such a forward model that can predict a representation of a future state, from the beginning of a navigation episode, if the episode were to be successful. Furthermore, in order for efficient training, we develop an algorithm to integrate a replay buffer into the model-free RL that alternates between training the policy and the forward model. We call our agent ForeSI; ForeSI is trained to imagine a future latent state that leads to success. By explicitly imagining such a state, during the navigation, our agent is able to take better actions leading to two main advantages: first, in the absence of an object detector, ForeSI presents a more robust policy, i.e., it leads to about 5% absolute improvement on the Success Rate (SR); second, when combined with an off-the-shelf object detector to help better distinguish the target object, our method leads to about 3% absolute improvement on the SR and about 2% absolute improvement on Success weighted by inverse Path Length (SPL), i.e., presents higher efficiency.
BibTeX
@inproceedings{moghaddam2022foresi,
  author = {Mohammad Mahdi Kazemi Moghaddam and Ehsan Abbasnejad and Qi Wu and Qinfeng (Javen) Shi and Anton van den Hengel},
  title = {{ForeSI: Success-Aware Visual Navigation Agent}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  doi = {10.1109/wacv51458.2022.00346},
}
Figure