Optimistic Agent: Accurate Graph-Based Value Estimation for More Successful Visual Navigation

Mohammad Mahdi Kazemi Moghaddam, Qi Wu, Ehsan Abbasnejad, Javen ShiOpenReview 2021 · DOI · OpenReview

Abstract

Humans possess a remarkable ability to search for target objects in unseen environments by leveraging prior knowledge and adapting to visual cues. Current Reinforcement Learning (RL) methods for visual navigation often lack this capability, frequently "giving up" when a target is not immediately visible. We propose the Optimistic Agent, which integrates externally learned prior knowledge of object co-locations into a neural graph. Central to our approach is the Graph-based Value Estimation (GVE) module, which provides a more accurate baseline for estimating the Advantage function in actor-critic RL. This module reduces value estimation error and enables convergence to a more optimal policy. Our agent maintains a realistic estimate of state value throughout navigation, leading to state-of-the-art results in the AI2THOR environment.

Introduction: The Problem of Early Capitulation

In visual navigation tasks, an agent must find a specific object (e.g., "toaster") in a previously unseen 3D environment. A major challenge in this domain is sparse rewards: if the agent does not see the target, it may perceive the current state as having zero value and stop prematurely—a behavior termed "early capitulation."

Humans avoid this by using common sense; seeing a fridge suggests a toaster is likely nearby. While previous research attempted to use Graph Neural Networks (GNNs) to provide this context, these methods often increased state-space complexity, making the RL problem harder to solve. The Optimistic Agent addresses this by decoupling knowledge from action selection, using a graph primarily to improve the critic's value estimation.

Methods: Graph-Based Value Estimation

The core of our architecture is the GVE module, which augments a standard Asynchronous Advantage Actor-Critic (A3C) framework.

The Optimistic Agent architecture integrates a Graph Transformer Network (GTN) and GVE module to refine value estimation based on object correlations.

Architecture Components

  1. Prior Knowledge Graph: We construct a graph where nodes represent objects and edges represent co-occurrence probabilities derived from the Visual Genome dataset. We utilize room-specific graph channels to account for different object relationships across contexts (e.g., a kitchen vs. a living room).
  2. Graph Transformer Networks (GTN): Rather than using a static graph, the GTN learns to update edge weights during training, allowing the agent to refine its internal "common sense" through experience.
  3. Value Function Decomposition: We decompose the value function into two parts: one based on the current visual state (via ResNet-18 and LSTM) and another based on graph-encoded correlations between visible objects and the target. This allows the critic to maintain "optimism" by recognizing that states near related objects have high potential value.
  4. Meta-Learning: We incorporate Model-Agnostic Meta-Learning (MAML) to allow the agent to quickly adapt its weights to the specific layout of a new environment at test time.

Key Results

The Optimistic Agent achieves new state-of-the-art performance on the AI2THOR benchmark, outperforming previous methods like SAVN in both success and efficiency.

Quantitative results showing state-of-the-art performance and qualitative trajectories comparing the Optimistic Agent (Ours) against the SAVN baseline.

Quantitative Performance

As shown in the results table, our full model (A3C+Graph+MAML+GVE) reaches a Success Rate (SR) of 43.8% and a Success weighted by Path Length (SPL) of 17.27. These improvements are particularly pronounced in long-range navigation tasks (Success > 5), where maintaining an accurate value estimate is most critical.

Qualitative Robustness

Qualitative analysis of navigation trajectories reveals that while baseline agents often stop and fail after a few steps when the target is not visible, the Optimistic Agent continues to navigate. By following "clue" objects identified through the GVE module, the agent maintains a high state value and successfully reaches the target. Quantitative error analysis confirms that the GVE module significantly reduces the L2 distance between predicted state values and ground-truth values compared to models without graph-based estimation.

Conclusion

By integrating structured prior knowledge into the value estimation process rather than the action space, the Optimistic Agent overcomes the hurdle of sparse rewards in visual navigation. This approach results in an agent that is more resilient, more efficient, and more capable of navigating complex, unseen environments.

Summary updated

Abstract

We humans can impeccably search for a target object, given its name only, even in an unseen environment. We argue that this ability is largely due to three main reasons: the incorporation of prior knowledge (or experience), the adaptation of it to the new environment using the observed visual cues and most importantly optimistically searching without giving up early. This is currently missing in the state-of-the-art visual navigation methods based on Reinforcement Learning (RL). In this paper, we propose to use externally learned prior knowledge of the relative object locations and integrate it into our model by constructing a neural graph. In order to efficiently incorporate the graph without increasing the state-space complexity, we propose Graph-based Value Estimation (GVE) module. GVE provides a more accurate baseline for estimating the Advantage function in actor-critic RL algorithm. This results in reduced value estimation error and, consequently, convergence to a more optimal policy. Through empirical studies, we show that our agent, dubbed as the optimistic agent, has a more realistic estimate of the state value during a navigation episode which leads to a higher success rate. Our extensive ablation studies show the efficacy of our simple method which achieves the state-of-the-art results measured by the conventional visual navigation metrics, e.g. Success Rate (SR) and Success weighted by Path Length (SPL), in AI2THOR environment.
BibTeX
@misc{moghaddam2021optimistic,
  author = {Mohammad Mahdi Kazemi Moghaddam and Qi Wu and Ehsan Abbasnejad and Javen Shi},
  title = {{Optimistic Agent: Accurate Graph-Based Value Estimation for More Successful Visual Navigation}},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00378},
  howpublished = {OpenReview},
}
Figure