Disentangling Foreground and Background for vision-Language Navigation via Online Augmentation

Yunbo Xu, Xuesong Zhang, Jia Li, Zhenzhen Hu, Richang HongarXiv 2025 · DOI · arXiv

Abstract

Vision-Language Navigation (VLN) agents often struggle with generalization in unseen environments due to overfitting on holistic visual features. We present COFA (Consensus-driven Online Feature Augmentation), a novel framework that disentangles visual inputs into foreground landmarks and background layouts. By employing a two-stage consensus voting mechanism, COFA dynamically selects the most relevant features for specific instructions. Our method achieves state-of-the-art performance on R2R and REVERIE benchmarks with minimal training overhead, demonstrating that selective attention to semantic and spatial cues is vital for robust navigation.

Introduction

Vision-Language Navigation (VLN) tasks require agents to interpret natural language instructions to navigate complex 3D environments. A persistent challenge in this field is generalization: agents frequently fail in "unseen" environments because they overfit to the specific visual textures of their training data. Traditional approaches attempt to mitigate this through massive offline data augmentation, which is computationally expensive and often ignores the structural composition of the visual scene.

Biologically, the human visual system processes foreground objects (semantic landmarks) and background layouts (spatial connectivity) through distinct pathways. For instance, a "corridor" provides essential spatial information for movement, while a "mug" serves as a critical semantic target for task completion. COFA leverages this biological inspiration to improve navigation by explicitly disentangling these features and selecting them based on task requirements.

Methodology: The COFA Framework

The COFA framework consists of two primary components: disentanglement and online augmentation.

Overview of the COFA framework, illustrating foreground/background disentanglement and the consensus voting pipeline.

1. Foreground and Background Disentanglement

The system first identifies and extracts relevant visual components:

  • Identification: An object detector (RAM++) identifies potential items, which are then filtered by a Large Vision-Language Model (Qwen2.5-VL) to retain "essential landmarks" pertinent to the room type.
  • Extraction: A text-driven segmentation model (EVF-SAM) generates binary masks for these landmarks, allowing the image to be split into "Foreground-only" and "Background-only" versions.
  • Encoding: These regions are converted into feature vectors using a CLIP visual encoder, alongside the original holistic view.

2. Consensus-driven Online Augmentation

Instead of providing all features simultaneously, COFA uses a two-stage voting mechanism to determine the most effective feature for a given viewpoint:

  • Stage 1 (Multi-agent Voting): Three agents, each specialized in original, foreground, or background features, vote on which feature type produces the most confident action prediction.
  • Stage 2 (Multi-trajectory Voting): Votes are aggregated across multiple trajectories passing through the same viewpoint to reach a robust consensus on the most reliable feature for that location.

Experiments and Results

We evaluated COFA on two major VLN benchmarks: Room-to-Room (R2R), which emphasizes path-following, and REVERIE, which focuses on remote object grounding.

Task-Specific Visual Preferences

A key finding of this work is that different navigation tasks exhibit distinct visual preferences. Analysis shows that agents performing the REVERIE task (finding objects) show a strong preference for foreground features. Conversely, in the R2R task (following paths), agents rely more heavily on background features to understand spatial layouts.

Distribution of feature preferences (Original, Foreground, Background) across R2R and REVERIE datasets.

Performance Benchmarks

COFA achieved state-of-the-art results across both benchmarks:

  • REVERIE: On the Val-Unseen split, COFA improved the Success Rate (SR) to 54.62% (up from a 46.98% baseline) and achieved a Success weighted by Path Length (SPL) of 38.17%.
  • R2R: The model reached a Success Rate of 74.2% in unseen environments, outperforming more complex offline augmentation methods.
  • Efficiency: Unlike offline methods, COFA performs augmentation at the feature level during training, incurring negligible architectural complexity or additional training time.

Conclusion and Future Work

COFA demonstrates that disentangling semantic landmarks from spatial layouts significantly enhances the generalization of VLN agents. While the reliance on background features aids navigation, it can occasionally impact precise object grounding upon arrival—a trade-off for future exploration. Additionally, the current pipeline depends on external pre-trained models like SAM and CLIP. Future research could focus on making the disentanglement process more end-to-end or adapting the voting mechanism for dynamic environments with moving objects.

Summary updated

Abstract

Following language instructions, vision-language navigation (VLN) agents are tasked with navigating unseen environments. While augmenting multifaceted visual representations has propelled advancements in VLN, the significance of foreground and background in visual observations remains underexplored. Intuitively, foreground regions provide semantic cues, whereas the background encompasses spatial connectivity information. Inspired on this insight, we propose a Consensus-driven Online Feature Augmentation strategy (COFA) with alternative foreground and background features to facilitate the navigable generalization. Specifically, we first leverage semantically-enhanced landmark identification to disentangle foreground and background as candidate augmented features. Subsequently, a consensus-driven online augmentation strategy encourages the agent to consolidate two-stage voting results on feature preferences according to diverse instructions and navigational locations. Experiments on REVERIE and R2R demonstrate that our online foreground-background augmentation boosts the generalization of baseline and attains state-of-the-art performance.
BibTeX
@misc{xu2025disentangling,
  author = {Yunbo Xu and Xuesong Zhang and Jia Li and Zhenzhen Hu and Richang Hong},
  title = {{Disentangling Foreground and Background for vision-Language Navigation via Online Augmentation}},
  year = {2025},
  eprint = {2510.00604},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.00604},
}
Figure