Abstract
VLN-Game is a zero-shot framework designed to enable robots to locate objects using complex natural language descriptions without task-specific training. By framing target identification as a game-theoretic equilibrium search between generative and discriminative vision-language models (VLMs), the system mitigates hallucinations and achieves state-of-the-art performance in both simulated and real-world environments.
Introduction: The Challenge of Specificity
Current Object Goal Navigation (ObjectNav) systems excel at identifying generic categories like "chair" or "table." However, they often falter when faced with specific instructions involving fine-grained attributes (e.g., "wooden," "black") or complex spatial relationships (e.g., "the chair between the whiteboard and the table").
While Large Language Models (LLMs) and Vision-Language Models (VLMs) offer significant reasoning capabilities, their application in robotics is hindered by "hallucinations"—where models confidently but incorrectly identify objects—and inconsistent responses based on prompting. VLN-Game addresses these reliability issues to enable high-stakes navigation in unknown 3D environments.
Methodology: The VLN-Game Pipeline
The VLN-Game framework follows a modular three-stage pipeline designed for robust zero-shot performance:
- 3D Object-Centric Mapping: As the robot explores, it constructs a 3D map where detected objects are stored as point clouds associated with CLIP (Contrastive Language-Image Pre-training) features. This persistent memory allows the robot to track seen objects and their locations.
- Frontier-Based Semantic Exploration: To avoid aimless wandering, the robot employs a "frontier" policy. It calculates similarity scores between the user's natural language instruction and the 3D map features to prioritize unexplored areas most likely to contain the target.
- Game-Theoretic Equilibrium Search: This core component treats object identification as a signaling game between two roles: a Generator (which selects the correct ID from candidates based on the description) and a Discriminator (which evaluates if a candidate image matches the description). Using the piKL algorithm, the system finds a Nash Equilibrium—a consensus between both roles—effectively filtering out VLM noise and hallucinations.
Experimental Evaluation
The framework was rigorously tested across several benchmarks to evaluate its navigation and grounding capabilities:
- Habitat-Matterport 3D (HM3D): In photo-realistic simulations, VLN-Game achieved a Success Rate (SR) of 0.613, surpassing existing zero-shot methods like VLFM (0.525) and InstructNav (0.580). It also demonstrated superior efficiency in Success weighted by Path Length (SPL).
- Goat-bench: On tasks involving complex spatial descriptions, VLN-Game (using GPT-4o) reached a 0.367 SR, significantly outperforming standard CLIP-based baselines (0.253).
- Real-world Deployment: Testing on a Jackal Robot platform confirmed the system's ability to handle noisy real-world sensor data and successfully navigate complex office layouts to find specific targets.
Key Insights and Future Directions
A primary takeaway from this research is that consensus-based decision-making is more reliable than single-model inference. By forcing a "Generator" and a "Discriminator" to reach an equilibrium, the system effectively double-checks its logic. Interestingly, smaller models using this equilibrium search can outperform much larger models using standard ranking, suggesting that the interaction framework is as critical as model scale.
Despite its success, VLN-Game faces challenges such as CLIP's sensitivity to viewpoints and a reliance on the underlying object detector's vocabulary. Future work will focus on integrating more advanced VLMs and extending the framework to lifelong navigation, where robots continuously update their environmental knowledge over extended periods.
Summary updated