L4D-Track: Language-to-4D Modeling Towards 6-DoF Tracking and Shape Reconstruction in 3D Point Cloud Stream

Jingtao Sun, Yaonan Wang, Mingtao Feng, Yulan Guo, Ajmal Mian, Mike Zheng Shou

Abstract

L4D-Track is a novel framework designed to achieve real-time, 6-DoF (6 Degrees of Freedom) tracking and 3D shape reconstruction from point cloud streams by leveraging natural language descriptions. By aligning 3D multimodal features with multi-level language captions, the model demonstrates remarkable "zero-shot" capabilities, successfully tracking and reconstructing objects from categories it has never encountered during training.

Introduction

Tracking the precise position and orientation (6-DoF pose) of an object while simultaneously reconstructing its 3D shape is a fundamental challenge in robotics and augmented reality. Traditional methods typically fall into two categories: instance-level methods, which require a precise CAD model of the object beforehand, or category-level methods, which are restricted to predefined categories such as "chair" or "car."

These systems often fail when encountering "out-of-distribution" objects. Furthermore, while 2D vision-language models (e.g., CLIP) have seen significant advancement, 3D-language modeling lags behind due to a severe shortage of paired 3D-text data. L4D-Track addresses this gap by using the semantic richness of language to generalize to unseen objects in real-time 3D video streams.

Methodology

L4D-Track introduces a "Language-to-4D" modeling paradigm composed of several innovative components:

Pairwise Implicit 3D Space Representation

Instead of direct pose regression, the model utilizes an Energy-based Pose Hypothesis to model movement uncertainty between frames. It also employs Neural Pose-aligned Fields (NPFs), which learn to reconstruct the object's shape in a way that is consistently aligned with its tracked pose across temporal sequences.

Real-time tracking and 3D shape reconstruction of a laptop in a point cloud stream.

GPT-assisted Language-to-4D Association

To overcome data scarcity, the authors back-projected 3D point clouds into 2D images and utilized BLIP-2 and GPT-4 to automatically generate high-quality, multi-level captions (video, scene, and object levels) for existing 3D datasets.

Multi-Level Multi-Modal Alignment

The core training involves a contrastive learning strategy that aligns features extracted from the 3D point cloud stream with generated text descriptions. This ensures that 3D features carry semantic meaning, allowing the model to identify and track new objects based on linguistic descriptions (e.g., "a yellow banana").

Experimental Results

The model was evaluated using the NOCS-REAL275 benchmark for category-level pose estimation and the YCB-Video dataset to test zero-shot generalization on unseen categories like a "master chef can" or "banana."

Performance Metrics

  • Superior Tracking: On NOCS-REAL275, L4D-Track outperformed the original NOCS baseline with a +45.9% gain in IoU75 and a +40.5% gain in the strict 5°2cm metric.
  • Zero-Shot Generalization: When tested on unseen categories in YCB-Video, L4D-Track outperformed previous state-of-the-art methods (e.g., CatTrack) by 20% to 30% in pose accuracy.
  • Efficiency: The system achieves approximately 20 FPS on an NVIDIA RTX A6000, meeting the requirements for real-world robotics applications.
Zero-shot 6-DoF tracking and reconstruction of a banana from the YCB-Video dataset.

Discussion and Conclusion

L4D-Track demonstrates that language can serve as a universal bridge for 3D understanding. By modeling pose changes as implicit distributions and distilling knowledge from 2D vision-language models, the framework achieves robust tracking and high-quality shape reconstruction. Future work will focus on expanding the diversity of the base training set to handle even more complex, wild shapes and reducing reliance on segmented masks.

Summary updated

Abstract

3D visual language multi-modal modeling plays an important role in actual human-computer interaction. However, the inaccessibility of large-scale 3D-language pairs restricts their applicability in real-world scenarios. In this paper, we aim to handle a real-time multi-task for 6-DoF pose tracking of unknown objects, leveraging 3D-language pre-training scheme from a series of 3D point cloud video streams, while simultaneously performing 3D shape reconstruction in current observation. To this end, we present a generic Language-to-4D modeling paradigm termed L4D-Track, that tackles zero-shot 6-DoF Tracking and shape reconstruction by learning pairwise implicit 3D representation and multi-level multi-modal alignment. Our method constitutes two core parts. 1) Pairwise Implicit 3D Space Representation, that establishes spatial-temporal to language coherence descriptions across continuous 3D point cloud video. 2) Language-to-4D Association and Contrastive Alignment, enables multi-modality semantic connections between 3D point cloud video and language. Our method trained exclusively on public NOCS-REAL275 dataset, achieves promising results on both two publicly benchmarks. This not only shows powerful generalization performance, but also proves its remarkable capability in zero-shot inference. The project is released at L4D- Track.
BibTeX
@inproceedings{sun2024l,
  author = {Jingtao Sun and Yaonan Wang and Mingtao Feng and Yulan Guo and Ajmal Mian and Mike Zheng Shou},
  title = {{L4D-Track: Language-to-4D Modeling Towards 6-DoF Tracking and Shape Reconstruction in 3D Point Cloud Stream}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01998},
}
Figure