UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing

Jiaxi Zhang, Yunheng Wang, Wei Lu, Taowen Wang, Weisheng Xu, Shuning Zhang, Yixiao Feng, Yuetong Fang, Renjing Xu

3D Visual Grounding (3DVG)—the ability for an AI to locate a specific object in a 3D scene based on a natural language description—is a "holy grail" for embodied AI. Whether it's a robot fetching "the half-empty mug on the corner of the mahogany desk" or an AR headset highlighting a specific valve for a technician, the system must bridge the gap between messy 3D geometry and nuanced human language.

TL;DR

UniGround is a training-free framework that achieves state-of-the-art zero-shot 3D visual grounding by decoupling geometric perception from semantic reasoning. By replacing supervised 3D detectors with a two-stage "parsing-then-reasoning" pipeline, it generalizes to any object in any environment—even those completely different from standard research benchmarks.

What Problem Does This Paper Address?

Most current 3DVG systems suffer from a "hidden" dependency. While they claim to be "zero-shot" or "open-vocabulary" because they use Large Language Models (LLMs), their perception front-ends are usually pre-trained on specific 3D datasets (like ScanNet).

This creates two major bottlenecks:

  1. Limited Generalization: If the 3D detector was trained only on chairs and tables, it might fail to "see" a specialized piece of medical equipment or a unique piece of art in a real-world setting.
  2. The "Seeing" Gap: Providing a Vision-Language Model (VLM) with just a single 2D crop of an object often deprives it of the spatial context (e.g., "to the left of the door") needed to resolve ambiguity.

UniGround aims to move away from domain-specific 3D models entirely, creating a system that can walk into a brand-new room and understand it immediately without further training.

Methods

The authors propose a two-stage pipeline that relies on off-the-shelf foundation models rather than task-specific training.

Stage 1: Global Candidate Filtering

Instead of using a 3D object detector, UniGround uses 2D-to-3D lifting.

  • Geometry: It partitions the 3D point cloud into "superpoints" (small, geometrically consistent clusters).
  • Topology: It uses the Segment Anything Model (SAM) on 2D camera views to find object boundaries and merges superpoints that consistently appear together across multiple views.
  • Semantics: It uses a Perception Encoder to generate semantic embeddings for these candidates, filtering the scene down to the top-$N$ most likely matches for the user's query.

Stage 2: Local Precision Grounding

Once candidates are identified, a VLM (like GPT-4o or GPT-5) performs the final "detective work" using a structured reasoning chain:

  • Spatial Relationship Prompts: The system generates "orbit renderings"—views that rotate around the scene center—to give the VLM a global perspective of the room's layout.
  • Candidate Visual Evidence: It provides high-resolution, multi-view crops of the specific candidate objects.
  • Chain-of-Thought (CoT) Reasoning: The VLM follows a protocol: Name the candidates $\rightarrow$ Parse the spatial relations in the query $\rightarrow$ Infer the target $\rightarrow$ Perform a "closed-loop" check to ensure the choice is consistent with the visual evidence.

Datasets

The researchers evaluated UniGround on:

  • ScanRefer: The standard benchmark for 3DVG based on the ScanNet dataset.
  • EmbodiedScan: A more challenging, diverse dataset (utilizing ARKitScenes) used to test how well the model transfers to new environments.
  • Real-World Benchmark: The authors collected their own data in four distinct environments (Office, Lounge, Corridor, Conference) to test the system under "wild" conditions like sensor noise and reconstruction artifacts.

Key Insights and Ideas

  • Decoupling is Power: By separating geometric parsing (finding "blobs" in 3D) from semantic reasoning (deciding which "blob" is the target), the system avoids the biases inherent in supervised 3D detectors.
  • The Bottleneck is Visual, Not Cognitive: The authors argue that modern VLMs are smart enough to do 3DVG; they just usually aren't given the right visual information. Providing both "global orbit" views and "local detail" views is crucial.
  • Training-Free > Supervised (Sometimes): In out-of-distribution (OOD) scenarios, UniGround actually outperforms models that were heavily trained on specific datasets because it hasn't "overfit" to the quirks of those datasets.

Key Results

  • New Zero-Shot SOTA: On ScanRefer, UniGround achieved 46.1% [email protected], a significant jump over previous open-world methods.
  • Superior Generalization: On the EmbodiedScan dataset, it reached 28.7% [email protected], outperforming the runner-up (LLM-G) by nearly 30% and even beating some fully supervised models.
  • Real-World Robustness: In live tests, UniGround maintained a 30-40% higher success rate than existing baselines like SeeGround and SeqVLM, which often collapsed when faced with real-world sensor noise.
  • Ablation Importance: Removing the "Candidate Visual Evidence" (local crops) caused accuracy to plummet from 56.2% to 33.3%, proving that global views alone aren't enough for fine-grained identification.

Limitations and Future Work

  • Inference Latency: Because the system relies on multiple VLM calls and high-resolution image processing, it is currently too slow for real-time robotic reaction.
  • Token Consumption: The multi-view prompting strategy uses a high number of tokens, which can be expensive when using commercial API-based models.
  • VLM Ceiling: The system's "intelligence" is ultimately capped by the reasoning capabilities of the underlying VLM. As models like GPT-5 or specialized embodied VLMs improve, UniGround will likely scale with them.

The authors hope this "reasoning-centric" approach inspires a shift away from expensive 3D data annotation toward more generalizable, training-free architectures.

Summary updated

Abstract

Understanding and localizing objects in complex 3D environments from natural language descriptions, known as 3D Visual Grounding (3DVG), is a foundational challenge in embodied AI, with broad implications for robotics, augmented reality, and human-machine interaction. Large-scale pre-trained foundation models have driven significant progress on this front, enabling open-vocabulary 3DVG that allows systems to locate arbitrary objects in a given scene. However, their reliance on pre-trained models constrains 3D perception and reasoning within the inherited knowledge boundaries, resulting in limited generalization to unseen spatial relationships and poor robustness to out-of-distribution scenes. In this paper, we replace this constrained perception with training-free visual and geometric reasoning, thereby unlocking open-world 3DVG that enables the localization of any object in any scene beyond the training data. Specifically, the proposed UniGround operates in two stages: a Global Candidate Filtering stage that constructs scene candidates through training-free 3D topology and multi-view semantic encoding, and a Local Precision Grounding stage that leverages multi-scale visual prompting and structured reasoning to precisely identify the target object. Experiments on ScanRefer and EmbodiedScan show that UniGround achieves 46.1\%/34.1\% [email protected]/0.5 on ScanRefer and 28.7\% [email protected] on EmbodiedScan, establishing a new state-of-the-art among zero-shot methods on EmbodiedScan without any 3D supervision. We further evaluate UniGround in real-world environments under uncontrolled reconstruction conditions and substantial domain shift, showing training-free reasoning generalizes robustly beyond curated benchmarks.
BibTeX
@misc{zhang2026uniground,
  author = {Jiaxi Zhang and Yunheng Wang and Wei Lu and Taowen Wang and Weisheng Xu and Shuning Zhang and Yixiao Feng and Yuetong Fang and Renjing Xu},
  title = {{UniGround: Universal 3D Visual Grounding via Training-Free Scene Parsing}},
  year = {2026},
  eprint = {2603.08131},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.08131},
}
Figure