POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images

Antonín Vobecký, Oriane Siméoni, David Hurych, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Josef Sivic

Abstract

POP-3D is a novel framework designed to predict 3D semantic voxel occupancy from 2D images without the need for manual 3D annotations. By leveraging a tri-modal self-supervised learning strategy that combines 2D images, LiDAR geometry, and pre-trained vision-language models (CLIP), POP-3D enables "open-vocabulary" 3D perception. This capability allows autonomous systems to identify and retrieve arbitrary objects—such as "stairs," "fire hydrants," or "strollers"—using only natural language queries, overcoming the limitations of traditional closed-vocabulary models.

Introduction

Understanding a 3D environment requires solving two fundamental challenges: determining geometry (where objects are) and semantics (what those objects are). Traditional semantic 3D occupancy prediction models are typically "closed-vocabulary," restricted to recognizing a small, pre-defined set of classes (e.g., "car," "pedestrian") established during manual labeling.

Manual 3D annotation is notoriously expensive and time-consuming. Furthermore, closed-vocabulary models are effectively blind to any object outside their training set. POP-3D addresses these issues by removing the requirement for 3D manual labels and enabling the model to understand any concept described in natural language, a capability known as open-vocabulary learning.

Overview of the POP-3D method and examples of open-vocabulary tasks like zero-shot occupancy segmentation and text-based 3D retrieval.

Methods

The POP-3D framework consists of a specific model architecture and a tri-modal training scheme designed for self-supervision.

Architecture

The model utilizes a 2D-3D encoder that takes surround-view images and "lifts" 2D features into a 3D voxel grid. This grid is then processed by two parallel heads:

  • Occupancy Head: A class-agnostic head that predicts whether a voxel is occupied or empty.
  • 3D-Language Head: This head predicts a high-dimensional language embedding for every occupied voxel. These embeddings are aligned with the mathematical space of CLIP text embeddings.

Tri-modal Self-Supervised Training

To avoid manual labels, the training process integrates three data sources:

  • Geometry (LiDAR): Raw, unlabeled LiDAR point clouds provide ground truth for the Occupancy Head. A voxel is considered occupied if a LiDAR point falls within it and empty if a ray passes through it.
  • Semantics (Vision-Language Models): The model uses MaskCLIP+ to extract semantic features from input images.
  • Alignment: 2D MaskCLIP+ features are projected into 3D space using LiDAR points, serving as a "teacher" for the 3D-Language Head. This allows the model to learn 3D representations based on 2D CLIP observations.

Crucially, at inference time, the LiDAR is discarded; the model performs 3D occupancy and semantic prediction using only 2D images.

The POP-3D architecture for open-vocabulary 3D occupancy prediction and the tri-modal self-supervised training pipeline.

Evaluation and Results

The model was evaluated on the nuScenes dataset and a newly introduced Open-Vocabulary Retrieval Benchmark containing 105 samples annotated with free-form language queries (e.g., "black hatchback," "fire hydrant").

Key Results

  • Occupancy Prediction: POP-3D achieved a class-agnostic IoU of 37.7, significantly outperforming the supervised TPVFormer baseline (26.2), demonstrating the effectiveness of self-supervision from raw LiDAR for learning geometry.
  • Zero-Shot Segmentation: On standard nuScenes classes, POP-3D reached approximately 78% of the performance of a fully supervised model (16.7 mIoU vs. 21.3 mIoU) without any manual labels.
  • 3D Retrieval: On the language-driven benchmark, POP-3D achieved 18.4 mAP, surpassing the MaskCLIP+ baseline of 14.9 mAP.

Discussion and Insights

A significant finding of this research is that 3D consistency improves 2D features. By forcing features to be consistent across multiple camera views within a 3D voxel grid, POP-3D filters out 2D noise, often outperforming its "teacher" model (MaskCLIP+). Additionally, decoupling geometry and semantics through a class-agnostic occupancy head makes the open-vocabulary task more robust.

Limitations and Future Work

The current model's spatial resolution (100x100x8 voxel grid) is relatively coarse, making it difficult to detect very small objects. Furthermore, the architecture processes single time steps; incorporating temporal information from video sequences could improve consistency and resolve occlusions. Finally, the model's semantic intelligence is inherently linked to the pre-trained vision-language model it distills from, meaning future improvements in 2D foundation models will directly benefit POP-3D.

Summary updated

BibTeX
@inproceedings{vobecky2023pop,
  author = {Antonín Vobecký and Oriane Siméoni and David Hurych and Spyros Gidaris and Andrei Bursuc and Patrick Pérez and Josef Sivic},
  title = {{POP-3D: Open-Vocabulary 3D Occupancy Prediction from Images}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=eBXM62SqKY},
}
Figure