Cross-Modal Match for Language Conditioned 3D Object Grounding

Yachao Zhang, Runze Hu, Ronghui Li, Yanyun Qu, Yuan Xie, Xiu Li

Abstract

3D object grounding is the task of identifying a specific object within a 3D point cloud based on a natural language query. Current two-stage approaches often fail due to a lack of global scene context and a misalignment between visual and linguistic feature representations. We propose xM_Match, a novel framework that addresses these issues through Bird’s-Eye-View Global Information Embedding (BEV-GIE) and Cross-Modal Consistency Learning (xMCL). By projecting 3D scenes into a 2D horizontal plane and aligning visual features with label-word semantics, xM_Match achieves state-of-the-art results on the Nr3D and Sr3D benchmarks, with significant improvements in challenging scenarios involving multiple similar objects.

Introduction

Locating objects in 3D space using natural language (e.g., "the chair next to the window") requires a model to understand both fine-grained object features and broad spatial relationships. Traditional methods typically extract local features for object proposals independently, leading to two primary "mismatches":

  1. Local vs. Global View: Language descriptions often rely on global scene context that local visual encoders miss.
  2. Feature Space Mismatch: Visual and textual encoders often map data to disparate mathematical spaces, making direct matching difficult.

Methodology: The xM_Match Framework

The proposed xM_Match framework introduces three core components to bridge these gaps.

BEV-GIE: Capturing Global Context

To efficiently capture spatial relationships without the overhead of full 3D scene processing, xM_Match employs Bird’s-Eye-View Global Information Embedding (BEV-GIE). Given that over 80% of spatial descriptions in standard datasets refer to horizontal relationships, the model projects object proposals onto a 2D horizontal plane. A mini-Vision Transformer (ViT) then processes these projections to model long-range dependencies and spatial context across the entire room.

xMCL: Cross-Modal Consistency Learning

The Cross-Modal Consistency Learning (xMCL) module aligns the visual feature space with the linguistic "label word" space. By utilizing a memory bank of object category features, xMCL ensures that the visual encoding of an object (e.g., a "table") is mathematically close to the embedding of the word "table." This alignment simplifies the final matching task by reducing it to a comparison of vectors within a shared semantic space.

The detailed architecture of xM_Match, highlighting the BEV-GIE and xMCL modules within a Teacher-Student distillation framework.

Teacher-Student Distillation

The framework utilizes a distillation strategy where a "Teacher" network, trained with access to ground-truth object labels, guides a "Student" network. The student learns to find objects using only raw point clouds and language prompts, aided by a Global Distillation Loss that transfers scene-wide contextual knowledge from the teacher.

Experiments and Results

The model was evaluated on the Nr3D (Natural Reference 3D), Sr3D (Synthetic Reference 3D), and ScanRefer datasets.

Quantitative Performance

xM_Match sets a new state-of-the-art (SOTA) for 3D object grounding using ground-truth proposals. On the Nr3D dataset, it achieved an overall accuracy of 66.2%, outperforming the previous best (ViL3DRel) by 1.8%. On Sr3D, it reached 74.6% accuracy.

Comparison of xM_Match against existing methods on the Nr3D and Sr3D benchmarks.

Notably, the performance gains were most significant in "Hard" subsets—scenes containing multiple objects of the same category—where xM_Match showed improvements of 2.5% on Nr3D and 3.4% on Sr3D. This suggests that the global context provided by BEV-GIE is crucial for disambiguating similar objects.

Conclusion

xM_Match demonstrates that integrating global spatial context via BEV projections and enforcing cross-modal consistency significantly enhances 3D object grounding. While the BEV approach is highly effective for horizontal relationships, future work may explore ways to better incorporate vertical spatial information. Additionally, adapting these consistency modules for one-stage grounding models remains a promising direction for reducing computational complexity.

Summary updated

Abstract

Language conditioned 3D object grounding aims to find the object within the 3D scene mentioned by natural language descriptions, which mainly depends on the matching between visual and natural language. Considerable improvement in grounding performance is achieved by improving the multimodal fusion mechanism or bridging the gap between detection and matching. However, several mismatches are ignored, i.e., mismatch in local visual representation and global sentence representation, and mismatch in visual space and corresponding label word space. In this paper, we propose crossmodal match for 3D grounding from mitigating these mismatches perspective. Specifically, to match local visual features with the global description sentence, we propose BEV (Bird’s-eye-view) based global information embedding module. It projects multiple object proposal features into the BEV and the relations of different objects are accessed by the visual transformer which can model both positions and features with long-range dependencies. To circumvent the mismatch in feature spaces of different modalities, we propose crossmodal consistency learning. It performs cross-modal consistency constraints to convert the visual feature space into the label word feature space resulting in easier matching. Besides, we introduce label distillation loss and global distillation loss to drive these matches learning in a distillation way. We evaluate our method in mainstream evaluation settings on three datasets, and the results demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{zhang2024cross,
  author = {Yachao Zhang and Runze Hu and Ronghui Li and Yanyun Qu and Yuan Xie and Xiu Li},
  title = {{Cross-Modal Match for Language Conditioned 3D Object Grounding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i7.28566},
}
Figure