A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding

Zhenyang Liu, Sixiao Zheng, Siyu Chen, Cairong Zhao, Longfei Liang, Xiangyang Xue, Yanwei Fu

Introduction

The field of Open-Vocabulary 3D Visual Grounding is essential for the development of autonomous agents and robots capable of navigating and interacting with the physical world using natural language. The primary objective is to enable a system to locate specific objects within a 3D scene based on arbitrary text prompts. While current state-of-the-art methods have made significant strides by mapping 2D Contrastive Language-Image Pre-training (CLIP) features into 3D neural representations—such as Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting (3DGS)—they often struggle with complex, multi-object queries.

This paper introduces SpatialReasoner, a framework designed to overcome the limitations of existing "language fields" by integrating Large Language Model (LLM) reasoning with a hierarchical feature field. By decomposing complex spatial queries and leveraging fine-grained physical properties, SpatialReasoner enables precise localization in scenarios where traditional methods fail.

The Limitations of Current 3D Language Fields

Existing 3D grounding methods typically suffer from two major "blind spots":

  1. Semantic Ambiguity in Language Reasoning: Most models treat a complex query (e.g., "the book on the chair") as a single global embedding. This often results in the model searching for a hybrid "book-chair" feature or becoming confused when multiple instances of the same object class are present in a scene.
  2. Lack of Geometric Context: Current methods often lack a true understanding of 3D geometry and spatial relationships. They struggle to distinguish between identical objects (e.g., two identical red books) because they cannot effectively process the physical relationship between a target object and its surrounding "anchor" objects.

The SpatialReasoner Framework

SpatialReasoner addresses these challenges through a two-pronged strategy: "Think Carefully" and "Look Carefully."

1. "Think Carefully": LLM-Driven Decomposition

Instead of processing a query as an indivisible unit, SpatialReasoner employs a fine-tuned LLM (such as TinyLlama) to parse the instruction into a structured format. The LLM decomposes the query into three distinct components:

  • Target: The primary object to be located (e.g., "book").
  • Anchor: The reference object used for spatial context (e.g., "chair").
  • Spatial Relation: The specific geometric link between the target and the anchor (e.g., "supported by").

This decomposition allows the system to first identify the anchor, significantly narrowing the search space for the target and ensuring that the spatial context is explicitly considered.

2. "Look Carefully": Hierarchical Feature Field

To improve the precision of the 3D representation, the framework constructs a hierarchical feature field that incorporates more than just semantic embeddings:

  • Visual Property Enhancement: The model integrates opacity and color directly from the underlying neural representation. Opacity helps define clear structural boundaries, while color provides the fine-grained detail necessary to distinguish between semantically identical but visually distinct items.
  • Instance Field: By deprojecting Segment Anything Model (SAM) masks into 3D space, the framework generates an "instance field." This allows the model to treat objects as discrete entities rather than a continuous cloud of points.
  • Instance Graph and Verification: Finally, the system constructs a graph of candidate targets and anchors. It verifies the spatial relationship—categorized as Horizontal, Vertical, Support, or Allocentric—against the actual 3D geometry to confirm the correct match.

One of the framework's greatest strengths is that it is representation-agnostic; it can be seamlessly integrated with various 3D backends, including NeRF, Instant-NGP, and 3D Gaussian Splatting.

Experimental Results

The authors evaluated SpatialReasoner across several benchmarks, including LERF, Replica, and a newly introduced dataset, Re3D, which specifically tests spatial reasoning with multiple identical objects.

Key findings include:

  • Superior Accuracy: On the LERF benchmark, SpatialReasoner (using a 3DGS backbone) achieved a localization accuracy of 91.7%, a dramatic improvement over the original LERF (37.6%) and LangSplat (46.1%).
  • Robust Spatial Reasoning: On the Re3D dataset, the framework achieved a 64.8% mIoU, nearly doubling the performance of previous methods that lack explicit spatial reasoning components.
  • Computational Efficiency: Despite its complexity, the system remains efficient. When paired with 3D Gaussian Splatting, it maintains an activation time of only 0.08 seconds per view.

Conclusion and Future Work

SpatialReasoner demonstrates that 3D visual grounding is not merely a semantic matching problem but a multi-step reasoning task. By re-introducing physical properties like color and opacity and using LLMs to decompose spatial logic, the framework sets a new standard for open-vocabulary 3D localization.

Future research will focus on expanding the vocabulary of spatial relations beyond the current four categories and addressing the challenges of dynamic scenes where objects may move over time. Additionally, reducing the dependency on the LLM's initial parsing accuracy remains a priority for making the system more robust to messy, real-world language.

Summary updated

BibTeX
@inproceedings{liu2025neural,
  author = {Zhenyang Liu and Sixiao Zheng and Siyu Chen and Cairong Zhao and Longfei Liang and Xiangyang Xue and Yanwei Fu},
  title = {{A Neural Representation Framework with LLM-Driven Spatial Reasoning for Open-Vocabulary 3D Visual Grounding}},
  booktitle = {ACM Multimedia},
  year = {2025},
  doi = {10.1145/3746027.3754918},
}
Figure