Generalizing 3D Representations Across Spatial Domains

112 concepts114 papers1 proposal

When a machine learning model trained on indoor living rooms fails to recognize a street scene, the culprit is almost always appearance shift. Changes in lighting, texture, and sensor modalities create a visual gap that 2D deep learning models struggle to bridge.

However, physical space obeys universal rules. Whether inside a room or out on a highway, the laws of geometry—such as spatial consistency, topological continuity, and occlusion—remain invariant. This realization has driven a powerful shift in 3D computer vision: by decoupling geometric structure from surface appearance, models can achieve remarkable cross-domain generalization. As shown by Gaurav Sharma et al. in Large Scale Novel Object Discovery in 3D, models trained purely on 3D geometric relationships can discover entirely unseen object categories in novel environments without any site-specific training.

Understanding why geometry generalizes requires looking at how modern architectures isolate spatial primitives, leverage self-supervised structural priors, and parameterize space to transcend scale.

Decoupling Features from Global Coordinates

Early deep learning approaches on 3D data were often fragile because they relied on global coordinate systems or shape-specific spectral bases. If an object bent, deformed, or scaled, the underlying mathematical representation changed completely, preventing the model from transferring its knowledge to new shapes.

To solve this, researchers shifted focus toward local, coordinate-free geometric representations. In Geodesic Convolutional Neural Networks on Riemannian Manifolds, Jonathan Masci et al. introduced a local "patch" operator using geodesic polar coordinates. Because this convolution operates within the local curvature of the surface rather than a global 3D grid, the learned features are naturally invariant to isometric deformations. This allowed models trained on one class of 3D objects to generalize effectively to entirely different shapes.

Global Coordinate Dependence (Fragile)       Local Geodesic Patches (Robust)
          [ Global Axis ]                               [ Local Surface ]
               |                                              |
      (X_1, Y_1, Z_1) -> Overfits                    (r, theta) -> Invariant to
      (X_2, Y_2, Z_2)    to absolute pose            intrinsic bending & deformation

A similar intuition applies to volumetric data. Instead of processing massive, rigid 3D grids, Yangyan Li et al. proposed FPNN: Field Probing Neural Networks for 3D Data, which uses a set of trainable local sensors to probe sparse data. By learning localized structural primitives, the model bypasses global geometric variations, making pre-trained features highly transferable across diverse object categories.

By focusing on these localized spatial relationships rather than global coordinates, the underlying principles of alignment and registration become universal. For instance, Haowen Deng demonstrated in 3D Local Features for Direct Pairwise Registration that decoupling geometric structure from motion via local feature arithmetic allows models trained on synthetic data to generalize seamlessly to real-world sensor scans.

Self-Supervised Structural Priors

If local parameterization provides the mechanism for invariance, self-supervised pre-training provides the statistical foundation. By training models to solve pretext tasks based purely on spatial structure, networks learn the "grammar" of 3D space from massive, diverse datasets like DL3DV-10K, a large-scale scene dataset introduced by Kun Wan et al.

Two primary self-supervised paradigms have proven highly effective for cross-domain transfer:

  1. Point-Level Contrastive Learning: In PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding, Jiatao Gu demonstrated that matching view-invariant local points across different transformations forces the network to map out a domain-agnostic geometric space. This representation is so robust that features learned entirely from indoor room scans transfer directly to outdoor driving environments.
  2. Masked Shape Prediction: By intentionally removing patches of a 3D mesh or point cloud and forcing the network to reconstruct them, models are compelled to learn underlying topological rules. Frameworks like MeshMAE by Shanshan Zhao et al. and the masked shape prediction method developed by Li Jiang et al. in Self-Supervised Pre-Training with Masked Shape Prediction for 3D Scene Understanding prevent the network from "cheating" by memorizing specific coordinate layouts. Instead, the model learns universal structural priors that generalize across disparate real-world sensor setups.

Geometry as an Invariant Manifold

To achieve zero-shot generalization from 2D images to 3D representations, a model must learn to separate what an object looks like from how it is positioned in space.

Xianglei Xing et al. formalized this in Unsupervised Disentangling of Appearance and Geometry by Deformable Generator Network. by splitting an architecture into an appearance generator and a geometric warping network. Because the geometric latent vectors capture abstract spatial transformations rather than texture, they can transfer poses seamlessly across entirely different object categories (e.g., from human faces to animals).

When lifting 2D observations to 3D space, modeling the underlying manifold of valid shapes prevents the network from overfitting to dataset-specific textures. For example, Karl Holmquist et al. developed DiffPose: Multi-hypothesis Human Pose Estimation using Diffusion Models, which uses generative distributions to predict 3D poses. Because the model learns the valid geometric manifold of the human body rather than direct image-to-3D regressions, it generalizes zero-shot to complex outdoor environments.

2D Image Input ------> [ Generative Diffusion Model ] ------> Multi-Hypothesis 3D Poses
                          (Learns valid structural             (Robust to out-of-distribution
                           manifold, ignores texture)           environments and occlusions)

This structural robustness is also evident in view-synthesis and reconstruction tasks. Olivia Wiles showed in SynSin: End-to-End View Synthesis From a Single Image that projecting 2D features into a 3D point cloud representation allows a model to learn generalizable spatial rules that transfer across diverse environments without fine-tuning. Similarly, Shangzhe Wu et al. demonstrated in MagicPony: Learning Articulated 3D Animals in the Wild that combining self-supervised visual features with implicit-explicit shape representations allows models trained on natural images to generalize their geometric understanding to abstract domains like sculptures and toys.

Bridging the Scale and Density Gap

Perhaps the most challenging domain shift occurs when moving between completely different scales—such as transferring a model from room-scale indoor scans to city-scale outdoor environments—or across varying point densities (e.g., from dense photogrammetry to sparse LiDAR).

To bridge these gaps, modern architectures rely on scale-invariant parameterizations and relative coordinate systems.

In MVSNet: Depth Inference for Unstructured Multi-view Stereo, Yao Yao et al. demonstrated that building deep homography warps based on camera frustums allows multi-view stereo networks to generalize from indoor training sets to massive outdoor reconstructions. Because the network reasons about relative depth and geometric consistency along camera rays rather than absolute spatial scales, the underlying logic remains identical regardless of the environment.

This concept is extended to continuous scene representations in OccAnyScene: Towards Unified Indoor-Outdoor 3D Occupancy Predictio. By representing scenes as pixel-frustum-centered Gaussians and utilizing relative parameterization, the authors created a single model capable of predicting 3D semantic occupancy across both indoor rooms and outdoor streets simultaneously, suffering negligible performance degradation.

Indoor Room Scale (Meters)                 Outdoor Street Scale (Kilometers)
   [ Local Frustum Gaussian ]  =======>  [ Local Frustum Gaussian ]

   * Shared Mechanism: Relative parameterization along camera rays bypasses 
     absolute metric scale, allowing a single model to unify both domains.

On the data density front, Kangli Wang et al. developed AnyPcc: Compressing Any Point Cloud with a Single Universal Model, proving that a unified architecture can effectively compress data ranging from sparse LiDAR to dense reconstructions by exploiting shared geometric priors.

The ultimate validation of this scale-transcendence is found in robotic navigation. Joanne Truong demonstrated in IndoorSim-to-OutdoorReal: Learning to Navigate Outdoors without any Outdoor Experience that a quadruped robot trained exclusively on indoor 3D simulations can successfully navigate complex real-world outdoor environments zero-shot. By forcing the robot to rely on abstracted depth inputs, the model isolates the fundamental spatial logic of occlusion and traversability—proving that when it comes to generalizing across domains, geometry is indeed enough.

Key Foundational Work

Geodesic Convolutional Neural Networks on Riemannian Manifolds (2015) — contributed 2 concept elements to this direction.
Data-Driven 3D Primitives for Single Image Understanding (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_domain_geometric_generalization_cr_d32385d9,
  title = {cross-domain geometric generalization (cross-domain 3D generalization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-domain-geometric-generalization-cross-domain-3d-generalization/rdr_d32385d9689442968f9f7558cbdbaa74},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_domain_geometric_generalization_cr_d32385d9_v1,
  title = {cross-domain geometric generalization (cross-domain 3D generalization)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-domain-geometric-generalization-cross-domain-3d-generalization/rdr_d32385d9689442968f9f7558cbdbaa74/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake