Abstract
Standard deep reinforcement learning models for visual navigation often fail to generalize to novel environments due to their "black-box" nature. We present a situational fusion approach that adaptively switches between 25 diverse visual representations at the action level. This method achieves a 2x higher success rate than standard baselines in unseen environments and demonstrates significant robustness to sensor failure.
Introduction
Visual navigation in complex, indoor environments requires a diverse set of skills, ranging from low-level geometric perception to high-level semantic understanding. Most current models treat navigation as a monolithic "black-box" task, mapping raw pixels directly to actions. While effective in training, these models frequently overfit to specific visual features of the training scenes, leading to poor generalization in new homes.
To address this, we argue that a robot should leverage a collection of specialized visual skills. For instance, avoiding a wall requires depth perception, while locating a target requires semantic segmentation. We propose "Situational Fusion," a framework that merges these skills based on the immediate environmental context.
Methodology: Situational Fusion
The core of our approach is the shift from feature-level fusion to action-level fusion. Instead of concatenating visual features into a single vector, we train separate action predictors for 25 different visual tasks derived from the Taskonomy project.
Action-Level Fusion
Each task-specific "expert" (e.g., Surface Normals, Object Classification) proposes an action candidate. A situational-fusion network then analyzes the current RGB observation to assign weights to these candidates. This allows the agent to prioritize geometric cues in narrow spaces and semantic cues in open areas.
Inter-task Affinity Regularization
To ensure the model selects a diverse and complementary set of skills, we employ a mathematical regularizer based on "task affinity." This prevents the model from relying on redundant information, such as multiple similar depth maps, thereby improving its ability to handle novel scenarios.
Experimental Setup
We evaluated our model using the GibsonEnv, which features 62 high-quality 3D scans of real-world indoor spaces. The agent was tasked with navigating to the nearest instance of a specific object (e.g., "Find Chair"). Training was conducted on 28 environments, with testing performed on 14 completely unseen environments to measure zero-shot generalization.
Results and Discussion
Our situational fusion model significantly outperformed standard baselines:
- Generalization: The model achieved a 44.2% success rate in unseen environments, more than double the 20.9% success rate of an ImageNet-pretrained ResNet-50 baseline.
- Fusion Strategy: Simply concatenating all features (feature-level fusion) only yielded a 25.6% success rate, highlighting that the fusion mechanism is as critical as the data itself.
- Robustness: The model proved highly resilient. In "Representation Dropping" tests, it maintained a success rate above 30% even when 17 of the 25 visual inputs were removed.
The model learned to dynamically shift its reliance: prioritizing 3D geometric representations like Euclidean Distance and Surface Normal Estimation for obstacle avoidance, and shifting to semantic representations when searching for target objects in cluttered rooms.
Conclusion
Situational Fusion provides a robust and interpretable framework for visual navigation. By fusing specialized visual skills at the action level, robots can better generalize to the complexities of real-world homes. Future work will explore learning new representations on the fly and bridging the simulation-to-real gap on physical robotic platforms.
Summary updated