Situational Fusion of Visual Representation for Visual Navigation

William B. Shen, Danfei Xu, Yuke Zhu, Leonidas J. Guibas, Li Fei-Fei, Silvio SavareseICCV 2019 · OpenReview

Abstract

Standard deep reinforcement learning models for visual navigation often fail to generalize to novel environments due to their "black-box" nature. We present a situational fusion approach that adaptively switches between 25 diverse visual representations at the action level. This method achieves a 2x higher success rate than standard baselines in unseen environments and demonstrates significant robustness to sensor failure.

Introduction

Visual navigation in complex, indoor environments requires a diverse set of skills, ranging from low-level geometric perception to high-level semantic understanding. Most current models treat navigation as a monolithic "black-box" task, mapping raw pixels directly to actions. While effective in training, these models frequently overfit to specific visual features of the training scenes, leading to poor generalization in new homes.

Figure 1: Overview of diverse environments and the variety of visual tasks (e.g., depth, segmentation, layout) used for navigation.

To address this, we argue that a robot should leverage a collection of specialized visual skills. For instance, avoiding a wall requires depth perception, while locating a target requires semantic segmentation. We propose "Situational Fusion," a framework that merges these skills based on the immediate environmental context.

Methodology: Situational Fusion

The core of our approach is the shift from feature-level fusion to action-level fusion. Instead of concatenating visual features into a single vector, we train separate action predictors for 25 different visual tasks derived from the Taskonomy project.

Action-Level Fusion

Each task-specific "expert" (e.g., Surface Normals, Object Classification) proposes an action candidate. A situational-fusion network then analyzes the current RGB observation to assign weights to these candidates. This allows the agent to prioritize geometric cues in narrow spaces and semantic cues in open areas.

Figure 6: The Situational Fusion architecture, illustrating how visual representations are mapped to action candidates and weighted based on the current observation and task affinity.

Inter-task Affinity Regularization

To ensure the model selects a diverse and complementary set of skills, we employ a mathematical regularizer based on "task affinity." This prevents the model from relying on redundant information, such as multiple similar depth maps, thereby improving its ability to handle novel scenarios.

Experimental Setup

We evaluated our model using the GibsonEnv, which features 62 high-quality 3D scans of real-world indoor spaces. The agent was tasked with navigating to the nearest instance of a specific object (e.g., "Find Chair"). Training was conducted on 28 environments, with testing performed on 14 completely unseen environments to measure zero-shot generalization.

Results and Discussion

Our situational fusion model significantly outperformed standard baselines:

  • Generalization: The model achieved a 44.2% success rate in unseen environments, more than double the 20.9% success rate of an ImageNet-pretrained ResNet-50 baseline.
  • Fusion Strategy: Simply concatenating all features (feature-level fusion) only yielded a 25.6% success rate, highlighting that the fusion mechanism is as critical as the data itself.
  • Robustness: The model proved highly resilient. In "Representation Dropping" tests, it maintained a success rate above 30% even when 17 of the 25 visual inputs were removed.

The model learned to dynamically shift its reliance: prioritizing 3D geometric representations like Euclidean Distance and Surface Normal Estimation for obstacle avoidance, and shifting to semantic representations when searching for target objects in cluttered rooms.

Conclusion

Situational Fusion provides a robust and interpretable framework for visual navigation. By fusing specialized visual skills at the action level, robots can better generalize to the complexities of real-world homes. Future work will explore learning new representations on the fly and bridging the simulation-to-real gap on physical robotic platforms.

Summary updated

Abstract

A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities. For example, to "go to the nearest chair", the agent might need to identify a chair in a living room using semantics, follow along a hallway using vanishing point cues, and avoid obstacles using depth. Therefore, utilizing the appropriate visual perception abilities based on a situational understanding of the visual environment can empower these navigation models in unseen visual environments. We propose to train an agent to fuse a large set of visual representations that correspond to diverse visual perception abilities. To fully utilize each representation, we develop an action-level representation fusion scheme, which predicts an action candidate from each representation and adaptively consolidate these action candidates into the final action. Furthermore, we employ a data-driven inter-task affinity regularization to reduce redundancies and improve generalization. Our approach leads to a significantly improved performance in novel environments over ImageNet-pretrained baseline and other fusion methods.
BibTeX
@misc{shen2019situational,
  author = {William B. Shen and Danfei Xu and Yuke Zhu and Leonidas J. Guibas and Li Fei-Fei and Silvio Savarese},
  title = {{Situational Fusion of Visual Representation for Visual Navigation}},
  year = {2019},
  eprint = {1908.09073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.09073},
}
Figure