Traditional 3D object detection is a "closed-set" problem. Models are trained to recognize a handful of categories—typically cars, pedestrians, and cyclists—and are effectively blind to everything else. Expanding this vocabulary is notoriously difficult because 3D point cloud annotation is orders of magnitude more expensive and labor-intensive than 2D image labeling.
To break this bottleneck, researchers are shifting toward Open-Vocabulary 3D Object Detection (OV-3DOD). The core intuition is to decouple "localization" (finding where an object is) from "classification" (knowing what it is) by leveraging the massive semantic knowledge already captured by 2D Vision-Language Models (VLMs) like CLIP.
The 2D to 3D Semantic Bridge
The fundamental challenge in OV-3DOD is that 3D sensors (LiDAR or point clouds) provide precise geometry but sparse semantics, while 2D images provide rich semantics but lack 3D structure.
Researchers like Chenfeng Xu and Xiaobao Wei in OV-3DET demonstrate that we can use pre-trained 2D detectors as "teachers." By projecting 3D point clouds into 2D image planes, the system can "borrow" labels from a 2D VLM. If a 2D model identifies a "traffic cone" in a crop of an image, and we know which 3D points project into that crop, we can supervise the 3D model to associate those points with the text embedding for "traffic cone."
Discovering Novel Objects via Geometry
If a model only learns from labeled data, it will never propose a bounding box for a category it hasn't seen. To solve this, frameworks like CoDA, developed by Yang Cao and Dan Xu, use 3D geometric priors to discover "novel" objects.
The logic is simple: even if the model doesn't know what a "grand piano" is, the point cloud shows a distinct, protruding cluster of points that looks like an object. By using class-agnostic proposal networks, CoDA identifies these clusters and then uses cross-modal alignment to name them. Similarly, Adrian Chow and colleagues introduced OV-SCAN, which uses a 3D box-fitting optimization to refine these "discovered" boxes, ensuring they are spatially consistent even without human labels (OV-SCAN).
Contextual and Synthetic Augmentation
Another way to bridge the data gap is to create the data that doesn't exist. Chenming Zhu and Tai Wang proposed Object2Scene, which takes 3D objects from diverse CAD datasets and "copy-pastes" them into existing 3D scenes (Object2Scene). By placing a 3D model of a "vacuum cleaner" into a scan of a living room, the detector learns the object's geometry and its relationship to the environment simultaneously.
Training-Free and Zero-Shot Detection
The most recent frontier moves away from training altogether. Models like Zoo3D, presented by Andrey Lemeshko and Danila Rukhovich, represent a "training-free" approach (Zoo3D). These systems use 2D foundation models like SAM (Segment Anything Model) to find object boundaries in images and then lift those boundaries into 3D space using geometric clustering.
Similarly, Haoran Jiang and Hao Zhao developed DetAny3D, which uses monocular images to "prompt" 3D detections (DetAny3D). By fusing the spatial precision of SAM with the semantic depth of DINO and CLIP, these models can localize almost any object in the wild without ever seeing a 3D training annotation for that specific category.
From Labels to Latent Spaces
The shift across these 15 papers is a move from explicit labeling to latent alignment. Instead of teaching a model that "this specific box is a chair," researchers are teaching models to align 3D features with a universal semantic space shared by text and images.
Whether through hierarchical alignment in INHA (Pengkun Jiao et al., INHA) or the "feature space expansion" of AV3DOD (Haomeng Zhang et al., Auto-Vocabulary 3D Object Detection), the goal is the same: an autonomous system that can navigate a room and identify a "vintage record player" or a "potted monstera" as easily as it identifies a "chair," simply by understanding the language that describes the world.