Abstract
This paper addresses the challenges of sparse rewards and sample inefficiency in multi-goal visual navigation. We propose a novel framework incorporating an Inverse Dynamics Model (InvDM) for dense auxiliary training signals and a Multi-goal Co-learning (MgCl) strategy to leverage unintentional successes across different navigation targets. Evaluated in the AI2-THOR environment, our approach demonstrates significantly improved convergence speed and robustness compared to standard reinforcement learning baselines.
Introduction
Visual navigation—teaching an agent to find its way through an environment using only camera inputs—is notoriously difficult due to partial observability. The complexity increases in goal-dependent navigation, where an agent must learn to reach a series of different targets from random starting positions without retraining for each new goal.
Standard Deep Reinforcement Learning (DRL) methods typically struggle in these scenarios due to sparse rewards (learning only occurs upon reaching a goal) and sample inefficiency (experience gained while seeking one goal is often discarded if it doesn't lead to that specific target). This paper explores how task cooperation can mitigate these issues.
Methodology
The authors build upon the Asynchronous Advantage Actor-Critic (A3C) framework, introducing two key modules to enhance learning efficiency:
- Inverse Dynamics Model (InvDM): This auxiliary, self-supervised task requires the network to predict the action taken given the previous and current visual states. By forcing the agent to understand which actions cause specific visual changes, it develops a more sophisticated internal representation of state-goal associations. This provides a continuous, dense training signal that offsets the problem of sparse rewards.
- Multi-goal Co-learning (MgCl): This strategy treats "failed" trajectories as "happy accidents." If an agent seeking Goal A happens to pass through the location of Goal B, MgCl intercepts that path and treats it as a successful pseudo-trajectory for Goal B. This allows the agent to learn multiple goals cooperatively from a single episode.
Experimental Setup
The model was evaluated using AI2-THOR, a photo-realistic 3D simulation environment. Testing spanned four scene categories: Bathroom, Bedroom, Living room, and Kitchen. The agent received 84x84 first-person RGB images and operated within a discrete action space (move forward, turn left, turn right).
Key Results and Discussion
Ablation studies confirmed that both InvDM and MgCl contribute significantly to performance. InvDM drastically improved convergence speed, particularly in complex, cluttered environments like the Kitchen. MgCl was most effective during early training phases when random exploration is high; as the agent became more proficient and took direct routes, the frequency of "accidental" goal encounters naturally decreased.
As shown in the performance curves, the proposed model significantly outperformed state-of-the-art baselines like DQN, A3C, and Target-driven Visual Navigation (TdVN). Notably, the model demonstrated superior scalability; while the performance of the A3C baseline dropped sharply when the number of goals increased from 2 to 4, the proposed model maintained stable, high rewards.
Conclusion
By combining self-supervised auxiliary tasks with multi-goal sample augmentation, this research demonstrates that agents can learn complex navigation tasks more efficiently through cooperation. Future work may involve extending these methods to continuous action spaces and real-world robotic platforms, where the assumptions of grid-based movement and accidental goal discovery may need further refinement.
Summary updated