Abstract
Vision-Language Navigation (VLN) agents often struggle with generalization in unseen environments due to overfitting on holistic visual features. We present COFA (Consensus-driven Online Feature Augmentation), a novel framework that disentangles visual inputs into foreground landmarks and background layouts. By employing a two-stage consensus voting mechanism, COFA dynamically selects the most relevant features for specific instructions. Our method achieves state-of-the-art performance on R2R and REVERIE benchmarks with minimal training overhead, demonstrating that selective attention to semantic and spatial cues is vital for robust navigation.
Introduction
Vision-Language Navigation (VLN) tasks require agents to interpret natural language instructions to navigate complex 3D environments. A persistent challenge in this field is generalization: agents frequently fail in "unseen" environments because they overfit to the specific visual textures of their training data. Traditional approaches attempt to mitigate this through massive offline data augmentation, which is computationally expensive and often ignores the structural composition of the visual scene.
Biologically, the human visual system processes foreground objects (semantic landmarks) and background layouts (spatial connectivity) through distinct pathways. For instance, a "corridor" provides essential spatial information for movement, while a "mug" serves as a critical semantic target for task completion. COFA leverages this biological inspiration to improve navigation by explicitly disentangling these features and selecting them based on task requirements.
Methodology: The COFA Framework
The COFA framework consists of two primary components: disentanglement and online augmentation.
1. Foreground and Background Disentanglement
The system first identifies and extracts relevant visual components:
- Identification: An object detector (RAM++) identifies potential items, which are then filtered by a Large Vision-Language Model (Qwen2.5-VL) to retain "essential landmarks" pertinent to the room type.
- Extraction: A text-driven segmentation model (EVF-SAM) generates binary masks for these landmarks, allowing the image to be split into "Foreground-only" and "Background-only" versions.
- Encoding: These regions are converted into feature vectors using a CLIP visual encoder, alongside the original holistic view.
2. Consensus-driven Online Augmentation
Instead of providing all features simultaneously, COFA uses a two-stage voting mechanism to determine the most effective feature for a given viewpoint:
- Stage 1 (Multi-agent Voting): Three agents, each specialized in original, foreground, or background features, vote on which feature type produces the most confident action prediction.
- Stage 2 (Multi-trajectory Voting): Votes are aggregated across multiple trajectories passing through the same viewpoint to reach a robust consensus on the most reliable feature for that location.
Experiments and Results
We evaluated COFA on two major VLN benchmarks: Room-to-Room (R2R), which emphasizes path-following, and REVERIE, which focuses on remote object grounding.
Task-Specific Visual Preferences
A key finding of this work is that different navigation tasks exhibit distinct visual preferences. Analysis shows that agents performing the REVERIE task (finding objects) show a strong preference for foreground features. Conversely, in the R2R task (following paths), agents rely more heavily on background features to understand spatial layouts.
Performance Benchmarks
COFA achieved state-of-the-art results across both benchmarks:
- REVERIE: On the Val-Unseen split, COFA improved the Success Rate (SR) to 54.62% (up from a 46.98% baseline) and achieved a Success weighted by Path Length (SPL) of 38.17%.
- R2R: The model reached a Success Rate of 74.2% in unseen environments, outperforming more complex offline augmentation methods.
- Efficiency: Unlike offline methods, COFA performs augmentation at the feature level during training, incurring negligible architectural complexity or additional training time.
Conclusion and Future Work
COFA demonstrates that disentangling semantic landmarks from spatial layouts significantly enhances the generalization of VLN agents. While the reliance on background features aids navigation, it can occasionally impact precise object grounding upon arrival—a trade-off for future exploration. Additionally, the current pipeline depends on external pre-trained models like SAM and CLIP. Future research could focus on making the disentanglement process more end-to-end or adapting the voting mechanism for dynamic environments with moving objects.
Summary updated