Abstract
Humans possess a remarkable ability to search for target objects in unseen environments by leveraging prior knowledge and adapting to visual cues. Current Reinforcement Learning (RL) methods for visual navigation often lack this capability, frequently "giving up" when a target is not immediately visible. We propose the Optimistic Agent, which integrates externally learned prior knowledge of object co-locations into a neural graph. Central to our approach is the Graph-based Value Estimation (GVE) module, which provides a more accurate baseline for estimating the Advantage function in actor-critic RL. This module reduces value estimation error and enables convergence to a more optimal policy. Our agent maintains a realistic estimate of state value throughout navigation, leading to state-of-the-art results in the AI2THOR environment.
Introduction: The Problem of Early Capitulation
In visual navigation tasks, an agent must find a specific object (e.g., "toaster") in a previously unseen 3D environment. A major challenge in this domain is sparse rewards: if the agent does not see the target, it may perceive the current state as having zero value and stop prematurely—a behavior termed "early capitulation."
Humans avoid this by using common sense; seeing a fridge suggests a toaster is likely nearby. While previous research attempted to use Graph Neural Networks (GNNs) to provide this context, these methods often increased state-space complexity, making the RL problem harder to solve. The Optimistic Agent addresses this by decoupling knowledge from action selection, using a graph primarily to improve the critic's value estimation.
Methods: Graph-Based Value Estimation
The core of our architecture is the GVE module, which augments a standard Asynchronous Advantage Actor-Critic (A3C) framework.
Architecture Components
- Prior Knowledge Graph: We construct a graph where nodes represent objects and edges represent co-occurrence probabilities derived from the Visual Genome dataset. We utilize room-specific graph channels to account for different object relationships across contexts (e.g., a kitchen vs. a living room).
- Graph Transformer Networks (GTN): Rather than using a static graph, the GTN learns to update edge weights during training, allowing the agent to refine its internal "common sense" through experience.
- Value Function Decomposition: We decompose the value function into two parts: one based on the current visual state (via ResNet-18 and LSTM) and another based on graph-encoded correlations between visible objects and the target. This allows the critic to maintain "optimism" by recognizing that states near related objects have high potential value.
- Meta-Learning: We incorporate Model-Agnostic Meta-Learning (MAML) to allow the agent to quickly adapt its weights to the specific layout of a new environment at test time.
Key Results
The Optimistic Agent achieves new state-of-the-art performance on the AI2THOR benchmark, outperforming previous methods like SAVN in both success and efficiency.
Quantitative Performance
As shown in the results table, our full model (A3C+Graph+MAML+GVE) reaches a Success Rate (SR) of 43.8% and a Success weighted by Path Length (SPL) of 17.27. These improvements are particularly pronounced in long-range navigation tasks (Success > 5), where maintaining an accurate value estimate is most critical.
Qualitative Robustness
Qualitative analysis of navigation trajectories reveals that while baseline agents often stop and fail after a few steps when the target is not visible, the Optimistic Agent continues to navigate. By following "clue" objects identified through the GVE module, the agent maintains a high state value and successfully reaches the target. Quantitative error analysis confirms that the GVE module significantly reduces the L2 distance between predicted state values and ground-truth values compared to models without graph-based estimation.
Conclusion
By integrating structured prior knowledge into the value estimation process rather than the action space, the Optimistic Agent overcomes the hurdle of sparse rewards in visual navigation. This approach results in an agent that is more resilient, more efficient, and more capable of navigating complex, unseen environments.
Summary updated