Abstract
Vision-and-Language Navigation (VLN) requires agents to follow natural language instructions in complex 3D environments. Current contrastive learning approaches often rely on coarse-grained negative samples, failing to teach agents the subtle visual-linguistic nuances necessary for high-precision navigation. We present the Fine-Grained VLN (FGVLN) framework, which utilizes Bayesian Optimization (BO) to generate challenging "hard" negative visual trajectories. By employing an adversarial training loop, FGVLN identifies and replaces critical visual frames to confuse the agent, forcing it to learn more robust and fine-grained alignments. Our method achieves state-of-the-art performance on the R2R and REVERIE benchmarks, particularly in unseen environments.
Introduction: The Fine-Grained Gap in VLN
Vision-and-Language Navigation (VLN) is a fundamental task in embodied AI, where a robot must navigate a realistic environment based on natural language instructions. Success depends on the agent's ability to align linguistic cues (e.g., "stop in front of the sofa") with visual observations.
Most existing models are trained using contrastive learning, which pulls "positive" pairs (correct instruction and path) together and pushes "negative" pairs apart. However, these negatives are typically "coarse-grained"—they consist of random paths or instructions that are easily distinguishable from the truth. This creates a "fine-grained gap": the agent never learns to distinguish between very similar scenarios, such as turning at the first versus the second door.
Methodology: Adversarial Bayesian Optimization
To bridge this gap, we introduce FGVLN, a framework that surgically creates "hard" negatives using Bayesian Optimization (BO).
1. Tree-structured Parzen Estimator (TPE)
Instead of random shuffling, we use a TPE-based BO model to identify "impactful" frames within a positive visual trajectory $V^+$. The BO sampler searches for a mask $M$ that selects frames which, if replaced, would most likely lead the navigation agent to make an error.
2. Inner Maximization and Outer Minimization
The training process follows an adversarial bi-level optimization:
- Inner Maximization: The BO sampler identifies the model's weak spots by generating fine-grained negatives that maximize the agent's ranking loss.
- Outer Minimization: The navigation agent (an online Multilayer Transformer) is trained to minimize this loss, learning to correctly identify the positive path even when presented with these highly confusing alternatives.
3. Training Stability
To ensure stable convergence, we implement a "target model" strategy. The BO sampler optimizes against a frozen version of the agent, which is only updated every $J$ steps. This prevents the "moving target" problem common in adversarial training.
Experimental Results
We evaluated FGVLN on the Room-to-Room (R2R) and REVERIE benchmarks.
Performance on Unseen Environments
FGVLN demonstrated significant gains in generalization to previously unseen 3D environments. On the R2R benchmark, our model achieved a 1.48% improvement in Success Rate (SR) and a 3.12% improvement in Success weighted by Path Length (SPL) over state-of-the-art baselines like Lily. On the REVERIE object-grounding task, FGVLN reached a Success Rate of 28.71%, outperforming the Airbert baseline.
Embedding Analysis
Statistical analysis of the learned encoders revealed that FGVLN produces more discriminative embeddings. By measuring the $L2$ distance between positive and fine-grained negative trajectories, we confirmed that our training method forces the model to attend to subtle visual cues, resulting in a more uniform and comprehensive processing of the visual sequence rather than relying on a few obvious landmarks.
Conclusion
The FGVLN framework demonstrates that the quality of negative samples is paramount for high-precision navigation. By using Bayesian Optimization to find and exploit a model's visual "blind spots," we can train agents that are significantly more robust and capable of handling the fine-grained nuances of real-world instructions. Future work will explore the use of generative models to create even more precise synthetic visual negatives.
Summary updated