Learning 3D Geometry through Multi-View Synthesis

This direction is merged into another direction: geometric self-supervision. Its last version stays readable and citable here.

Version 1 of 120 papers24 conceptsfrozen 2026-10-04

How can a machine learn the concept of "volume" or "symmetry" just by looking at flat images? Traditionally, teaching a model about the 3D world required expensive 3D CAD models or manual annotations of 6D poses. However, a growing body of research suggests that the laws of geometry and perspective themselves can serve as a supervisory signal. By tasking a model with predicting what an object looks like from a new, unseen angle—a process often called novel view synthesis—researchers are forcing neural networks to internalize 3D structural regularities without ever seeing a 3D label.

The Intuition: Prediction Requires Understanding

The core insight behind this direction is that successful view synthesis is impossible without a latent understanding of 3D structure. If a model is given a single image of a chair and asked to predict what that chair looks like from behind, it cannot simply memorize pixels. To succeed, it must "discover" that objects have depth, that they are often symmetric, and that certain parts (like legs) support others.

In ShapeCodes, Dinesh Jayaraman, Kristen Grauman, and Ruohan Gao demonstrate this by "lifting" a single 2D image into a "viewgrid"—a complete set of images from all possible angles. By forcing the latent space to represent the object from every side, the model naturally learns to capture geometric primitives. Similarly, Georg Poier and Horst Bischof argue in Being lazy at labelling for pose estimation that the laws of perspective provide enough supervision to train complex models for 3D hand pose estimation with 100x less labeled data.

Lifting 2D Features into 3D Space

A recurring mechanism in this field is the explicit or implicit "lifting" of 2D image features into a 3D representation, such as a voxel grid or a point cloud, and then projecting them back down to a 2D plane to check for accuracy.

In VoxDet, Bowen Li and colleagues use a reconstruction objective where the model must generate novel 2D views from a latent 3D voxel structure. This "reconstruct-to-predict" loop ensures that the internal 3D representation is spatially accurate. CrossJEPA, developed by Ranga Rodrigo and Mohamed Afham, takes this further by conditioning a predictor on camera pose. By asking the model "what would the 3D point cloud look like from this specific camera coordinate?", they distill 2D knowledge into a 3D encoder in a highly efficient, self-supervised manner.

Geometry as a Masking Problem

Recent breakthroughs in generative modeling have reframed 3D consistency as a sequence prediction or inpainting task. If you mask out a portion of a scene, the model must use its knowledge of 3D symmetry and occlusion to fill it in correctly.

  • Inpainting for Consistency: Gene Chou and Kai Zhang introduced KFC-W, which uses multiview inpainting with high masking ratios. To fill the "holes" in unposed internet photos, the model is forced to exploit geometric priors, resulting in 3D-consistent video generation without any explicit camera data.
  • Sequence Modeling: ViewMask-1-to-3 by Rui Zhu and Ping Luo treats multi-view generation as a discrete diffusion task. They found that training on multi-view sequences leads to emergent capabilities, such as the ability to rotate objects or complete backgrounds without specific training for those tasks.

From Synthesis to Downstream Tasks

The ultimate proof of this approach lies in its application to traditional computer vision problems like pose estimation and gaze tracking. Instead of training on $(Image, Pose)$ pairs, these models train on $(Image A, Image B, Relative Transform)$ triplets.

For example, MURAUER uses "view prediction" as a proxy task to learn hand poses from unlabeled real-world data. By predicting an alternative camera perspective, the model learns features that are inherently pose-relevant. Similarly, NeRF-Pose by Fu Li and Benjamin Busam uses Neural Radiance Fields to first reconstruct an object's implicit geometry from weak 2D labels. This reconstructed volume then provides the "ground truth" to train a pose regression network, effectively allowing the model to supervise itself.

Even in human-centric tasks like gaze following, the same principle applies. Following Gaze Across Views by Adrià Recasens and Antonio Torralba demonstrates that by optimizing for a gaze target across disparate camera views, a network implicitly learns to reconcile different coordinate systems and 3D orientations.

Conclusion

The shift toward multi-view synthesis represents a move away from "teaching by labeling" and toward "teaching by physics." By requiring models to maintain consistency across different perspectives, researchers are leveraging the inherent structure of the physical world to build more robust, 3D-aware artificial intelligence. Whether through voxels, diffusion, or radiance fields, the message is clear: to understand the world in 3D, a model must first learn to imagine it from every angle.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learning_3d_geometry_through_multi_view__ed1fde92,
  title = {Learning 3D Geometry through Multi-View Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-3d-geometry-through-multi-view-synthesis/rdr_ed1fde92095941d0b3ad00984e83a767},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_3d_geometry_through_multi_view__ed1fde92_v1,
  title = {Learning 3D Geometry through Multi-View Synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-3d-geometry-through-multi-view-synthesis/rdr_ed1fde92095941d0b3ad00984e83a767/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}