VLN-Game: Vision-Language Equilibrium Search for Zero-Shot Semantic Navigation

Hamidreza Kasaei, Bangguo Yu, Yuzhen Liu, Lei Han, Hamidreza Kasaei 0001, Tingguang Li, Ming Cao

Abstract

VLN-Game is a zero-shot framework designed to enable robots to locate objects using complex natural language descriptions without task-specific training. By framing target identification as a game-theoretic equilibrium search between generative and discriminative vision-language models (VLMs), the system mitigates hallucinations and achieves state-of-the-art performance in both simulated and real-world environments.

Introduction: The Challenge of Specificity

Current Object Goal Navigation (ObjectNav) systems excel at identifying generic categories like "chair" or "table." However, they often falter when faced with specific instructions involving fine-grained attributes (e.g., "wooden," "black") or complex spatial relationships (e.g., "the chair between the whiteboard and the table").

While Large Language Models (LLMs) and Vision-Language Models (VLMs) offer significant reasoning capabilities, their application in robotics is hindered by "hallucinations"—where models confidently but incorrectly identify objects—and inconsistent responses based on prompting. VLN-Game addresses these reliability issues to enable high-stakes navigation in unknown 3D environments.

Figure 1: VLN-Game enables a robot to find specific objects, such as a "black chair between a whiteboard and a table," by leveraging game-theoretic vision-language models.

Methodology: The VLN-Game Pipeline

The VLN-Game framework follows a modular three-stage pipeline designed for robust zero-shot performance:

  1. 3D Object-Centric Mapping: As the robot explores, it constructs a 3D map where detected objects are stored as point clouds associated with CLIP (Contrastive Language-Image Pre-training) features. This persistent memory allows the robot to track seen objects and their locations.
  2. Frontier-Based Semantic Exploration: To avoid aimless wandering, the robot employs a "frontier" policy. It calculates similarity scores between the user's natural language instruction and the 3D map features to prioritize unexplored areas most likely to contain the target.
  3. Game-Theoretic Equilibrium Search: This core component treats object identification as a signaling game between two roles: a Generator (which selects the correct ID from candidates based on the description) and a Discriminator (which evaluates if a candidate image matches the description). Using the piKL algorithm, the system finds a Nash Equilibrium—a consensus between both roles—effectively filtering out VLM noise and hallucinations.
Figure 2: The VLN-Game architecture integrates 3D mapping, semantic exploration, and a game-theoretic equilibrium search for robust target identification.

Experimental Evaluation

The framework was rigorously tested across several benchmarks to evaluate its navigation and grounding capabilities:

  • Habitat-Matterport 3D (HM3D): In photo-realistic simulations, VLN-Game achieved a Success Rate (SR) of 0.613, surpassing existing zero-shot methods like VLFM (0.525) and InstructNav (0.580). It also demonstrated superior efficiency in Success weighted by Path Length (SPL).
  • Goat-bench: On tasks involving complex spatial descriptions, VLN-Game (using GPT-4o) reached a 0.367 SR, significantly outperforming standard CLIP-based baselines (0.253).
  • Real-world Deployment: Testing on a Jackal Robot platform confirmed the system's ability to handle noisy real-world sensor data and successfully navigate complex office layouts to find specific targets.

Key Insights and Future Directions

A primary takeaway from this research is that consensus-based decision-making is more reliable than single-model inference. By forcing a "Generator" and a "Discriminator" to reach an equilibrium, the system effectively double-checks its logic. Interestingly, smaller models using this equilibrium search can outperform much larger models using standard ranking, suggesting that the interaction framework is as critical as model scale.

Despite its success, VLN-Game faces challenges such as CLIP's sensitivity to viewpoints and a reliance on the underlying object detector's vocabulary. Future work will focus on integrating more advanced VLMs and extending the framework to lifelong navigation, where robots continuously update their environmental knowledge over extended periods.

Summary updated

Abstract

Following human instructions to explore and search for a specified target in an unfamiliar environment is a crucial skill for mobile service robots. Most of the previous works on object goal navigation have typically focused on a single input modality as the target, which may lead to limited consideration of language descriptions containing detailed attributes and spatial relationships. To address this limitation, we propose VLN-Game, a novel zero-shot framework for visual target navigation that can process object names and descriptive language targets effectively. To be more precise, our approach constructs a 3D object-centric spatial map by integrating pre-trained visual-language features with a 3D reconstruction of the physical environment. Then, the framework identifies the most promising areas to explore in search of potential target candidates. A game-theoretic vision language model is employed to determine which target best matches the given language description. Experiments conducted on the Habitat-Matterport 3D (HM3D) dataset demonstrate that the proposed framework achieves state-of-the-art performance in both object goal navigation and language-based navigation tasks. Moreover, we show that VLN-Game can be easily deployed on real-world robots. The success of VLN-Game highlights the promising potential of using game-theoretic methods with compact vision-language models to advance decision-making capabilities in robotic systems. The supplementary video and code can be accessed via the following link: https://sites.google.com/view/vln-game.
BibTeX
@article{kasaei2026vln,
  author = {Hamidreza Kasaei and Bangguo Yu and Yuzhen Liu and Lei Han and Hamidreza Kasaei and Tingguang Li and Ming Cao},
  title = {{VLN-Game: Vision-Language Equilibrium Search for Zero-Shot Semantic Navigation}},
  journal = {IEEE Trans. Robotics},
  year = {2026},
  doi = {10.1109/tro.2026.3677047},
}
Figure