Deep learning thrives on massive datasets, but in the 3D world, ground-truth data is notoriously hard to come by. While we can easily snap billions of 2D photos, capturing their 3D counterparts—such as watertight meshes, dense voxel grids, or precise signed distance functions—requires specialized, expensive hardware like LiDAR or meticulous manual annotation. As noted by Gkioxari et al. in Mesh R-CNN, this dependency on dense 3D ground truth severely restricts our ability to scale models to diverse real-world object categories.
Even when 3D data is collected via physical sensors, it is often noisy, sparse, or incomplete. Traditional supervised frameworks struggle here because they expect pristine, closed geometries that rarely exist in the wild (Liao et al., Deep Marching Cubes: Learning Explicit Surface Representations). For organic categories like animals (Wu et al., MagicPony: Learning Articulated 3D Animals in the Wild) or dynamic phenomena like fluid flows (Franz et al., Learning to Estimate Single-View Volumetric Flow Motions without 3D Supervision), obtaining ground-truth 3D annotations is technically prohibitive.
To scale 3D computer vision, researchers have shifted away from direct 3D supervision. Instead, they design architectures that learn 3D structure implicitly by leveraging the rich, abundant information available in 2D images and geometric consistency.
Differentiable Rendering as a Supervisory Bridge
If we cannot provide a neural network with a target 3D shape to copy, how can it learn to output 3D? The answer lies in differentiable rendering.
Imagine a network that takes a 2D image and predicts a 3D mesh. Without 3D ground truth, we cannot compute a loss directly on the mesh vertices. However, if we pass that predicted mesh through a rendering engine that simulates how light interacts with geometry, we can project the 3D shape back into a 2D image. Because the rendering process is mathematically differentiable, we can compute a loss by comparing this rendered image against the original 2D observation. The gradients from this 2D loss flow backward through the renderer, directly modifying the 3D shape parameters.
This mechanism allows models to infer 3D shapes simply by ensuring their 2D projections match reality. For instance, Nguyen-Phuoc et al. demonstrated in RenderNet: A deep convolutional network for differentiable rendering from 3D shapes how a novel projection unit can enable end-to-end inverse graphics tasks. Similarly, Gadelha et al. introduced Projective GANs in 3D Shape Induction from 2D Views of Multiple Objects, which train a 3D generator using only 2D silhouettes. The model learns to generate realistic 3D shapes because its "shadows" must match the distribution of real 2D object silhouettes.
Multi-View and Cycle Consistency
While a single 2D view provides a powerful constraint, it suffers from inherent geometric ambiguity—many different 3D shapes can produce the exact same 2D projection. To resolve this, researchers leverage multi-view consistency.
The intuition is straightforward: if a predicted 3D model is accurate, it must explain the object's appearance from any viewpoint. As explored by Tulsiani et al. in Multi-View Consistency as Supervisory Signal for Learning Shape and Pose Prediction, observing an object from multiple angles allows a model to establish an "ecological supervision" signal. If the network predicts a shape from View A, that shape must remain consistent when projected into View B. Alternatively, Wiles et al. showed in 3D Surface Reconstruction by Pointillism that sparse point correspondences across different views can act as a powerful self-supervision signal to lift 2D images into dense 3D structures.
This concept can be extended to establish pixel-level correspondences across different instances of the same category without manual labels. In Learning Dense Correspondence via 3D-Guided Cycle Consistency, Zhou et al. utilized a "cycle consistency" loop. By mapping a pixel from a real image to a 3D CAD model, and then back to another real image, the network enforces that a closed-loop path must return to its exact starting point. The 3D structure acts as a geometric anchor, forcing the network to learn meaningful spatial relationships without explicit paired supervision.
Distribution Matching for Shape Completion
Another facet of the 3D ground truth shortage occurs in shape completion, where models must predict a full 3D shape from a partial scan. Normally, this requires pairs of partial and complete scans. However, acquiring such paired data in real-world environments is exceptionally difficult.
To circumvent this, researchers turn to adversarial distribution matching. Instead of forcing a network to match a specific paired ground truth, they train a generator to map partial scans onto a learned manifold of complete shapes. As shown by Chen et al. in Unpaired Point Cloud Completion on Real Scans using Adversarial Training, a discriminator can evaluate whether the completed shape looks like a plausible member of a clean shape dataset, without requiring an exact one-to-one match during training. This allows the model to learn the underlying geometric priors of completeness from unpaired, real-world scans.
Similarly, Zhang in Probabilistic Implicit Scene Completion leverages continuous generative frameworks to produce diverse, plausible completions, bypassing the need for deterministic paired targets that often lead to blurry, averaged reconstructions. By framing the problem as distribution matching rather than pixel-perfect regression, models successfully bypass the need for direct 3D supervision.