Abstract
Text-to-point-cloud (T2P) localization aims to infer precise spatial positions within 3D point cloud maps from natural language descriptions, reflecting how humans perceive and communicate spatial layouts through language. However, existing methods largely rely on shallow text-point cloud correspondence without effective spatial reasoning, limiting their accuracy in complex environments. To address this limitation, we propose VLM-Loc, a framework that leverages the spatial reasoning capability of large vision-language models (VLMs) for T2P localization. Specifically, we transform point clouds into bird's-eye-view (BEV) images and scene graphs that jointly encode geometric and semantic context, providing structured inputs for the VLM to learn cross-modal representations bridging linguistic and spatial semantics. On top of these representations, we introduce a partial node assignment mechanism that explicitly associates textual cues with scene graph nodes, enabling interpretable spatial reasoning for accurate localization. To facilitate systematic evaluation across diverse scenes, we present CityLoc, a benchmark built from multi-source point clouds for fine-grained T2P localization. Experiments on CityLoc demonstrate VLM-Loc achieves superior accuracy and robustness compared to state-of-the-art methods.
Introduction
Imagine waiting for an autonomous robotaxi in a dense urban canyon. GPS signals often degrade due to multipath interference from skyscrapers, making precise pickup difficult. In such scenarios, a human might tell the car: "I am on the gray road, east of a gray sidewalk, south of a black vegetation..." This task, known as Text-to-Point-Cloud (T2P) localization, requires a machine to translate natural language into precise 3D coordinates.
While humans perform this spatial grounding instinctively, previous machine learning approaches have struggled. Earlier methods relied on "shallow" matching—identifying similar features in text and 3D data without a deep understanding of spatial relationships. These models often fail in complex environments where multiple objects share similar appearances or when descriptions involve relative spatial logic. VLM-Loc addresses this by shifting the paradigm from simple pattern matching to high-level spatial reasoning powered by Large Vision-Language Models (VLMs).
Methodology
The VLM-Loc framework transforms the unstructured 3D point cloud into a format that leverages the pre-trained spatial intelligence of VLMs. The pipeline consists of three primary components: Dual Map Representation, Partial Node Assignment, and Autoregressive Decoding.
Dual Map Representation
To bridge the gap between 3D point clouds and 2D-centric VLMs, the system converts the map into two complementary formats:
- Bird’s-Eye-View (BEV) Images: A top-down, colorized 2D projection that captures dense geometric layouts and global context.
- Scene Graphs: A structured representation where nodes represent objects (e.g., "building," "traffic light") containing semantic labels and precise pixel coordinates.
Partial Node Assignment (PNA)
A core innovation of VLM-Loc is the Partial Node Assignment mechanism. Rather than directly predicting coordinates, the model is trained to explicitly associate textual hints (e.g., "the green tree") with specific nodes in the scene graph. This mechanism is designed to handle "partial" matches, allowing the model to recognize when a described object is present in the map, missing, or located outside the current frame. This intermediate reasoning step ensures that the final localization is grounded in verifiable object relationships.
Training and Inference
By fine-tuning general-purpose VLMs using Low-Rank Adaptation (LoRA), the framework leverages the massive spatial knowledge these models acquired during pre-training. During inference, the system generates a structured JSON response. This output includes the node assignments—providing an interpretable "chain of thought"—and the final estimated 2D coordinates.
The CityLoc Benchmark
To evaluate T2P localization rigorously, we introduced CityLoc, a multi-source benchmark:
- CityLoc-K: Derived from the KITTI-360 LiDAR dataset, featuring complex urban scenes with diverse semantic categories.
- CityLoc-C: Derived from CityRefer (UAV-based photogrammetry). This dataset is used to test cross-domain generalization, evaluating if a model trained on LiDAR data can successfully localize within maps generated from drone imagery in entirely different cities.
Experimental Results
VLM-Loc demonstrates significant improvements over existing state-of-the-art (SOTA) methods like CMMLoc:
- Accuracy: On the CityLoc-K test set, VLM-Loc achieved a 35.91% Recall@5m, representing a 14.20% absolute improvement over previous benchmarks.
- Generalization: When tested on the unseen CityLoc-C dataset, VLM-Loc maintained its performance lead, nearly doubling the accuracy of baseline models and proving its robustness across different sensor types and urban layouts.
- Interpretability: Qualitative analysis shows the model correctly grounds textual hints to map objects even in cluttered scenes, often achieving localization errors of less than 1 meter.
- Scaling: Performance scales positively with model size; larger backbones (e.g., 32B parameters) significantly outperform smaller variants (2B or 8B), suggesting that T2P localization benefits from the increased "intelligence" of larger VLMs.
Discussion and Future Work
While VLM-Loc sets a new standard for accuracy and interpretability, challenges remain regarding inference speed. Currently operating at approximately 0.23 to 0.36 frames per second, the system is suitable for "one-shot" initial localization but requires further optimization for real-time high-speed navigation.
Future research directions include handling more conversational and ambiguous human language beyond template-based descriptions and developing "active" localization agents. Such agents could potentially move or rotate sensors to better align the machine's perspective with a human's verbal description, further closing the gap between human and machine spatial communication.
Summary updated