Category-Agnostic Priors for Universal 3D Understanding

This direction is split into other directions: zero-shot geometric generalization. Its last version stays readable and citable here.

Version 1 of 1495 papers543 conceptsfrozen 2026-10-04

Traditional 3D computer vision has often relied on a "dictionary" approach: to reconstruct a chair, a model must first know what a chair is. By training on specific categories, models learn the average proportions of a seat or the typical placement of legs. However, this approach fails the moment the model encounters a novel object—a piece of modern art, a custom industrial part, or a complex protein.

A growing body of research is shifting toward category-agnostic geometric priors. Instead of memorizing what specific objects look like, these models learn the underlying "grammar" of 3D space—how surfaces curve, how parts connect, and how 2D images map to 3D volumes regardless of the object's label.

Decomposing Objects into Geometric Primitives

The core intuition behind category-agnosticism is that while global shapes are diverse, the local building blocks of objects are often shared. A table leg and a chair leg share similar cylindrical geometry; a car hood and a computer monitor both involve planar surfaces.

Xingyi Zhou and colleagues demonstrated this with StarMap StarMap for Category-Agnostic Keypoint and Viewpoint Estimation, which maps keypoints to a canonical 3D space rather than using rigid category labels. By focusing on these shared structures, the model can estimate the viewpoint of an entirely unseen object category with the same accuracy as a known one. Similarly, Giorgia Pitteri and Slobodan Ilic developed CorNet CorNet: Generic 3D Corners for 6D Pose Estimation of New Objects without Retraining, which detects generic 3D corners. Because corners are universal geometric primitives, a robot trained on a few items can locate a completely new industrial part using only its CAD model.

This modularity is taken further by Jiayuan Gu and Kaichun Mo, who factorize 3D reconstruction into independent attributes like part orientation and connectivity Compositionally Generalizable 3D Structure Prediction. When a model learns how parts fit together generally, it can "hallucinate" the structure of a bed even if it was only ever trained on chairs.

Lifting 2D Knowledge to 3D Space

3D data is notoriously scarce compared to the trillions of 2D images available. To build universal priors, researchers are "lifting" 2D visual intelligence into the third dimension.

Abhinav Kumar and Justin C. Johnson introduced Omni3D Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild, a massive dataset that forces models to generalize across indoor and outdoor scenes. They found that models trained on such diverse data develop "virtual depth" transformations that work across different camera sensors. This suggests that exposure to enough 2D variety allows a model to internalize universal geometric relationships.

Taking this to the extreme, Xingyu Miao and Deli Zhao proposed a pipeline to lift massive 2D datasets like COCO into metric-accurate 3D representations Towards Scalable Spatial Intelligence via 2D-to-3D Data Lifting. By pre-training on these "lifted" datasets, models learn robust spatial priors and object size distributions that transfer zero-shot to real-world sensor data.

Generative Priors and Shape Completion

When a sensor only sees the front of an object, the model must infer the back. Instead of using a template, modern methods use generative models to "fill in" the most likely geometry based on a learned distribution of all possible 3D shapes.

Yuhan Li and Bingbing Ni developed 3DQD Generalized Deep 3D Shape Prior via Part-Discretized Diffusion Process, a unified shape prior using a diffusion process. Because it learns a part-based representation, it can perform tasks like point cloud completion or text-to-shape generation without task-specific fine-tuning.

In a similar vein, Tianxin Huang and Gim Hee Lee showed that pre-trained 2D diffusion models can be used to complete 3D point clouds zero-shot Zero-shot Point Cloud Completion Via 2D Priors. By leveraging the vast "visual common sense" embedded in 2D generative models, the system can hallucinate missing 3D regions for out-of-domain objects more robustly than supervised models.

Generalizing Beyond Visual Objects

The power of category-agnostic geometric modeling extends into specialized scientific domains. In drug discovery, Ming-Hsiu Wu and Degui Zhi demonstrated that explicitly modeling the 3D spatial interactions between proteins and ligands A Folding–Docking–Affinity framework for protein–ligand binding affinity prediction allows for better generalization to entirely new biological spaces. While sequence-based models often overfit to known patterns, spatial modeling captures the fundamental physics of how molecules fit together.

Why It Works: From Memorization to Reasoning

The shift toward category-agnosticism works because it moves the bottleneck from data labeling to geometric consistency. As Chao-Yuan Wu and Jitendra Malik argue in their work on multiview coding Multiview Compressive Coding for 3D Reconstruction, training on diverse, real-world data allows models to develop general spatial priors.

When a model is no longer tethered to a specific label like "chair," it is free to learn the universal laws of 3D existence: that objects have volume, that occlusions hide surfaces, and that perspective changes predictably with movement. This "spatial intelligence" is what enables a model to reconstruct the back of a toy it has never seen before, simply because it understands how 3D objects, in general, must behave.

Papers in this version

Showing 60 of 495 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_category_agnostic_priors_for_universal_3_7d3ef794,
  title = {Category-Agnostic Priors for Universal 3D Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/category-agnostic-priors-for-universal-3d-understanding/rdr_7d3ef794186244849eed24d84f62571f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_category_agnostic_priors_for_universal_3_7d3ef794_v1,
  title = {Category-Agnostic Priors for Universal 3D Understanding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/category-agnostic-priors-for-universal-3d-understanding/rdr_7d3ef794186244849eed24d84f62571f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}