Abstract
3D object grounding is the task of identifying a specific object within a 3D point cloud based on a natural language query. Current two-stage approaches often fail due to a lack of global scene context and a misalignment between visual and linguistic feature representations. We propose xM_Match, a novel framework that addresses these issues through Bird’s-Eye-View Global Information Embedding (BEV-GIE) and Cross-Modal Consistency Learning (xMCL). By projecting 3D scenes into a 2D horizontal plane and aligning visual features with label-word semantics, xM_Match achieves state-of-the-art results on the Nr3D and Sr3D benchmarks, with significant improvements in challenging scenarios involving multiple similar objects.
Introduction
Locating objects in 3D space using natural language (e.g., "the chair next to the window") requires a model to understand both fine-grained object features and broad spatial relationships. Traditional methods typically extract local features for object proposals independently, leading to two primary "mismatches":
- Local vs. Global View: Language descriptions often rely on global scene context that local visual encoders miss.
- Feature Space Mismatch: Visual and textual encoders often map data to disparate mathematical spaces, making direct matching difficult.
Methodology: The xM_Match Framework
The proposed xM_Match framework introduces three core components to bridge these gaps.
BEV-GIE: Capturing Global Context
To efficiently capture spatial relationships without the overhead of full 3D scene processing, xM_Match employs Bird’s-Eye-View Global Information Embedding (BEV-GIE). Given that over 80% of spatial descriptions in standard datasets refer to horizontal relationships, the model projects object proposals onto a 2D horizontal plane. A mini-Vision Transformer (ViT) then processes these projections to model long-range dependencies and spatial context across the entire room.
xMCL: Cross-Modal Consistency Learning
The Cross-Modal Consistency Learning (xMCL) module aligns the visual feature space with the linguistic "label word" space. By utilizing a memory bank of object category features, xMCL ensures that the visual encoding of an object (e.g., a "table") is mathematically close to the embedding of the word "table." This alignment simplifies the final matching task by reducing it to a comparison of vectors within a shared semantic space.
Teacher-Student Distillation
The framework utilizes a distillation strategy where a "Teacher" network, trained with access to ground-truth object labels, guides a "Student" network. The student learns to find objects using only raw point clouds and language prompts, aided by a Global Distillation Loss that transfers scene-wide contextual knowledge from the teacher.
Experiments and Results
The model was evaluated on the Nr3D (Natural Reference 3D), Sr3D (Synthetic Reference 3D), and ScanRefer datasets.
Quantitative Performance
xM_Match sets a new state-of-the-art (SOTA) for 3D object grounding using ground-truth proposals. On the Nr3D dataset, it achieved an overall accuracy of 66.2%, outperforming the previous best (ViL3DRel) by 1.8%. On Sr3D, it reached 74.6% accuracy.
Notably, the performance gains were most significant in "Hard" subsets—scenes containing multiple objects of the same category—where xM_Match showed improvements of 2.5% on Nr3D and 3.4% on Sr3D. This suggests that the global context provided by BEV-GIE is crucial for disambiguating similar objects.
Conclusion
xM_Match demonstrates that integrating global spatial context via BEV projections and enforcing cross-modal consistency significantly enhances 3D object grounding. While the BEV approach is highly effective for horizontal relationships, future work may explore ways to better incorporate vertical spatial information. Additionally, adapting these consistency modules for one-stage grounding models remains a promising direction for reducing computational complexity.
Summary updated