In 2D computer vision, we have a "universal currency": the pixel. Whether you are performing object detection, segmentation, or generative art, the data is almost always a regular grid of values. This standardization allowed the field to converge on dominant architectures like the ResNet or the Vision Transformer, which could then be scaled to massive datasets.
3D deep learning has no such currency. Instead, it suffers from a "fragmented landscape," as described by Edward J. Smith and colleagues in their work on Kaolin. If you want to represent a chair, you might use a dense grid of voxels, a sparse point cloud, a geometric mesh, or a continuous neural field. Each format has its own mathematical language, making it nearly impossible to use a model designed for one on another.
The research direction of Representation Agnostic 3D Deep Learning seeks to break these silos. The goal is to create a single, unified backbone that can process 3D information regardless of how it is stored.
The Bottleneck of Structural Diversity
The primary reason 3D hasn't had its "ChatGPT moment" yet, according to Chaoning Zhang and Chenghao Li, is this lack of a unified data representation. In When ChatGPT for Computer Vision Will Come? From 2D to 3D, they argue that the computational overhead of switching between fragmented formats like point clouds and voxels prevents the adoption of the scaling laws that transformed NLP.
This fragmentation creates several practical hurdles:
- Incompatible Benchmarks: As Anke Bellmann notes, it is difficult to perform "apples-to-apples" comparisons because different reconstruction algorithms produce different output types (e.g., depth maps vs. surface orientations).
- Task-Specific Silos: A model optimized for point clouds often cannot be applied to meshes without significant redesign, a problem highlighted in the survey by Jean Lahoud and Fahad Shahbaz Khan.
- Tooling Fragmentation: Even within a single sub-field like 3D Gaussian Splatting, Chongjie Ye points out that incompatible internal representations prevent the development of universal downstream tools for rendering or enhancement.
Strategies for Unification
To overcome this, researchers are pursuing three main strategies to build "representation-agnostic" systems.
1. Mathematical Abstraction
Instead of looking at the data structure (the "container"), these methods look at the underlying signal. Anpei Chen and Andreas Geiger introduced Factor Fields, which unifies coordinate-based MLPs, hash tables, and tensor decompositions into a single mathematical framework. By decomposing signals into "factors," they create a bridge between disparate neural representation methods.
Similarly, Arianna Rampini and Luca Cosmo found that by moving from 3D coordinates to the spectral domain (the vibration frequencies of a shape), they could create Universal Spectral Adversarial Attacks that fool classifiers regardless of whether the input is a mesh or a point cloud.
2. Alignment with 2D Foundation Models
Since 2D vision is already standardized, many researchers are "borrowing" 2D features to anchor 3D data. Junsheng Zhou and Xinlong Wang developed Uni3D, a billion-parameter model that aligns 3D point clouds with the feature space of 2D image-text models. This allows the 3D model to inherit the "common sense" of 2D foundation models, bypassing the need for a native 3D standard.
3. Representation-Agnostic Encoders
New architectures are being designed to treat 3D inputs as generic features rather than specific geometries. Animesh Karnewar and colleagues proposed GOEmbed, which uses "Gradient Origin Embeddings" to learn features that don't care if the underlying 3D structure is a voxel grid or a coordinate-based MLP. This allows a single encoding architecture to be used across diverse modeling formats.
Why It Works: The Intuition of Latent Space
The core intuition behind this research is that geometry is an emergent property, not a fixed format.
In Omnivore, Rohit Girdhar and Ishan Misra demonstrate that a single Vision Transformer can learn a shared representation for images, videos, and 3D depth maps. By treating all inputs as patches of information, the model learns that a "table" has the same semantic meaning whether it's a flat image or a 3D shape.
When we move toward representation-agnosticism, we shift the focus from how the data is stored to what the data represents. As Anita Hu shows with VoMP, once you have a representation-agnostic model, you can predict physical properties (like density or Young's modulus) for any 3D asset—be it a mesh, a NeRF, or a Gaussian Splat—in a single feed-forward pass.
By stripping away the constraints of specific data structures, the field is finally moving toward the "universal backbone" necessary to scale 3D intelligence to the level of modern language and vision models.