Unified Neural Architectures for Heterogeneous 3D Data

28 concepts30 papers3 proposals

In 2D computer vision, the pixel grid reigns supreme. Whether an image captures a medical scan or a natural landscape, it is formatted as a regular, dense grid of intensity values. This structural uniformity allowed the field to rapidly converge on standardized, highly scalable macro-architectures like ResNets and Vision Transformers.

In the 3D world, this structural unity shatters. As highlighted in broad surveys of the field, 3D data lacks a single native representation. Instead, researchers must navigate a fragmented landscape of unorganized point clouds, rigid volumetric voxel grids, topologically complex polygon meshes, continuous signed distance functions (SDFs), and neural radiance fields.

Each format demands a fundamentally different mathematical approach. Point clouds require permutation-invariant operators; voxels rely on sparse 3D convolutions; meshes depend on graph neural networks. As Yatian Pang et al. and Yuenan Hou et al. observe, this architectural fragmentation prevents 3D deep learning from easily adopting general-purpose self-supervised learning techniques or establishing a standardized backbone capable of generalizing across diverse spatial tasks.

To overcome this, a major research direction has emerged focused on building unified neural architectures and representation-agnostic frameworks that can bridge these heterogeneous formats.

Mapping Heterogeneous Formats to Shared Latent Spaces

The first core intuition for unifying 3D data is to decouple the input format from the underlying geometric intelligence. Even if a shape is captured as a voxel grid by one sensor and a point cloud by another, the intrinsic geometry of the object remains identical.

[Point Cloud] ───> (Point Encoder) ───┐
                                      v
[Voxel Grid]  ───> (Voxel Encoder) ───> [ Unified Latent Space ] ───> Universal Tasks
                                      ^
[Multi-View]  ───> (View Encoder)  ───┘

To exploit this, Sanjeev Muralikrishnan et al. introduced Shape Unicode, a framework that trains a shared latent space across voxels, point clouds, and multi-view images. By forcing format-specific encoders to map their outputs into a single continuous "Unicode" space, the model allows a network processing a single modality to benefit from features and geometric properties learned across all other modalities simultaneously.

Taking this representation agnosticism further, Xiaoyang Huang et al. proposed Representation-Agnostic Shape Fields (RASF). Instead of locking a neural network into a specific explicit structure, RASF abstracts the geometry into a continuous field representation. This shift allows a single downstream feature extraction pipeline to operate seamlessly regardless of whether the underlying asset originates as a mesh, a voxel grid, or a point cloud, bypassing traditional format-locked constraints.

Transcending Geometry via Implicit and Spectral Domains

Rather than trying to align irregular explicit structures like points and voxels, another powerful strategy shifts the problem entirely into coordinate-free or frequency-based domains.

One prominent approach leverages Implicit Neural Representations (INRs), where a 3D shape is parameterized not by explicit coordinates, but by the weights of a small neural network trained to represent its volume or surface. Luca De Luigi et al. developed inr2vec, a framework that treats these network weights directly as the input data for downstream tasks. By operating in the space of neural network parameters rather than explicit geometry, inr2vec creates a unified, resolution-independent pipeline for classification, retrieval, and generation that completely avoids explicit 3D reconstruction.

Alternatively, the fragmentation problem can be solved by translating shapes into the spectral domain. Arianna Rampini et al. demonstrated the power of this domain by designing universal adversarial attacks based on the intrinsic vibration frequencies (the eigenfunctions of the Laplace-Beltrami operator) of 3D shapes. Because these spectral properties depend on the underlying geometry rather than how it is discretized, perturbations computed in the spectral domain transfer seamlessly across different resolutions and entirely different formats, such as from polygon meshes to unorganized point clouds.

Scaling Foundation Models via 2D Alignment

To build true 3D foundation models, architectures must scale up significantly—a feat historically bottlenecked by the lack of standardized 3D backbones. To circumvent this, recent architectures anchor heterogeneous 3D data to the highly optimized, standardized scaling laws of 2D vision transformers.

For example, Junsheng Zhou et al. developed Uni3D, a 3D foundation model scaled to one billion parameters. Instead of inventing a new 3D architecture from scratch, the authors initialized a standard Vision Transformer (ViT) with pretrained 2D weights and aligned its tokenized 3D features with a shared image-text embedding space. This strategy allows the model to inherit the robust representation capacity of massive 2D datasets while remaining flexible enough to handle diverse 3D inputs.

Similarly, Rohit Girdhar et al. introduced Omnivore, a single vision transformer that ingests images, videos, and single-view 3D patches using the exact same parameters. While models like Omnivore demonstrate impressive cross-modal generalization, extending this unified processing to fully volumetric voxels or complex meshes remains an active frontier.

To systematically evaluate these emerging architectures, frameworks like Unifi3D, introduced by Nina Wiedemann et al., provide standardized benchmarking environments. By evaluating voxels, SDFs, and point clouds on a level playing field, these frameworks help researchers isolate whether performance bottlenecks stem from the neural architecture itself or the underlying data representation, paving the way toward a truly unified 3D paradigm.

Key Foundational Work

Shape Unicode: A Unified Shape Representation (2019) — contributed 1 concept elements to this direction.
Directional Histogram Model for Three-Dimensional Shape Similarity (2003) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_3d_representation_heterogeneity_fd406af8,
  title = {3D representation heterogeneity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/3d-representation-heterogeneity/rdr_fd406af80f104ef68e17ba49ed3f3783},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_3d_representation_heterogeneity_fd406af8_v1,
  title = {3D representation heterogeneity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/3d-representation-heterogeneity/rdr_fd406af80f104ef68e17ba49ed3f3783/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake