Standard 3D reconstruction algorithms—whether traditional pipelines like Structure-from-Motion (SfM) and Multi-View Stereo (MVS), or modern neural approaches like Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting—share a fundamental vulnerability: they are largely structure-blind. They treat the physical world as an unorganized collection of independent points, voxels, or continuous coordinate fields.
In human-made environments, this assumption breaks down. A blank white wall contains no distinct visual features for a matching algorithm to track, causing neural surfaces to "melt" or become jagged. Sensor noise and occlusions leave gaps that data-driven models struggle to fill logically. Yet, humans easily navigate these spaces because we operate with an internal library of geometric rules: walls are flat, floors are horizontal, corners meet at right angles, and objects are often symmetric.
To bridge this gap, a prominent research direction focuses on embedding geometric and structural regularities directly into the 3D reconstruction process. By treating these regularities as hard constraints or strong inductive biases rather than optional downstream properties, models can resolve visual ambiguities, eliminate long-range tracking drift, and synthesize physically plausible geometry from sparse or noisy data.
The Power of Planar Primitives
The simplest way to enforce structural regularity is to shift the fundamental unit of reconstruction from the isolated 3D point to the planar surface. In urban and indoor environments, the vast majority of observed points belong to a small number of flat planes.
Treating scenes as collections of planes drastically simplifies optimization and eliminates high-frequency noise. For instance, András Bódis-Szomorú et al. demonstrated in Fast, Approximate Piecewise-Planar Modeling Based on Sparse Structure-from-Motion and Superpixels that assuming a scene is piecewise-planar allows dense 3D models to be computed in seconds rather than hours, bypassing the heavy computational overhead of dense MVS.
Enforcing these regularities early in the tracking pipeline yields even greater stability. In visual-inertial odometry, Xin Li showed in Leveraging Planar Regularities for Point Line Visual-Inertial Odometry that binding points and lines to shared co-planar surfaces prevents tracking errors from degrading the map, keeping naturally flat surfaces smooth. Similarly, Seongdo Kim 0001 et al. developed a system in Multi-planar Monocular Reconstruction of Manhattan Indoor Scenes that optimizes plane equations directly within the Bundle Adjustment step. Instead of adjusting millions of independent points, the system refines a handful of structural planes, resulting in clean, structurally coherent indoor maps.
[Noisy Point Cloud] ──(Structure-Blind Optimization)──> [Jagged, Melted Surfaces]
[Noisy Point Cloud] ──(Planar & Co-Planarity Priors)──> [Clean, Watertight Planes]
However, fitting planes independently can still lead to misalignments at boundaries. To solve this, Mulin Yu et al. proposed a global optimization framework in Finding Good Configurations of Planar Primitives in Unorganized Point Clouds. By evaluating configurations using a multi-objective energy function that explicitly penalizes non-parallelism and non-orthogonality between adjacent surfaces, their method ensures that individual primitives snap together into perfectly aligned, architecturally accurate models.
Anchoring Geometry with the Manhattan-World Assumption
Beyond individual planes, human-made architecture typically conforms to a global coordinate system where major surfaces are aligned with three mutually orthogonal axes. This is known as the Manhattan-world assumption.
Enforcing a global Manhattan frame provides a powerful anchor against long-range tracking drift. In SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels, Jianxiong Xiao utilized horizontal floor planes and orthogonal wall structures within a generalized bundle adjustment framework to successfully eliminate the warping and accumulation errors that typically plague large-scale indoor mapping.
When applied to modern neural implicit representations, Manhattan constraints solve the problem of textureless regions. For example, Haoyu Guo et al. introduced Neural 3D Scene Reconstruction with the Manhattan-world Assumption (ManhattanSDF), which applies orthogonal and planar regularizers to Signed Distance Fields (SDFs). Where a standard neural network would produce arbitrary geometry due to a lack of visual gradients (such as on a blank white wall), the Manhattan prior forces the network to interpret the region as a perfectly flat surface perpendicular to the floor.
Without Manhattan Prior: [Camera View] ──> (Textureless Wall) ──> [Ambiguous/Melted SDF]
With Manhattan Prior: [Camera View] ──> (Textureless Wall) ──> [Enforced 90° Alignment] ──> [Perfect Flat Surface]
To make these systems practical, the orientation of the Manhattan frame must be discovered automatically. Nikola Popovic et al. addressed this in Surface Normal Clustering for Implicit Representation of Manhattan Scenes by introducing a self-supervised surface normal clustering module. By grouping predicted normals into orthogonal clusters during training, the model dynamically learns the room's layout without requiring manual semantic annotations.
This structural alignment is equally effective for novel view synthesis from extreme angles. In CylinderSplat: 3D Gaussian Splatting with Cylindrical Triplanes for Panoramic Novel View Synthesis, Wang Qiwei et al. designed a cylindrical triplane representation that inherently aligns with the vertical walls and flat floors of indoor environments, allowing the model to accurately predict missing geometry in unobserved areas from a single panoramic image. Furthermore, when visual data is severely restricted or highly incomplete, Sangwoo Ryu et al. demonstrated in Planar Abstraction and Inverse Rendering of 3D Indoor Environments that enforcing right-angle intersections allows a system to logically "hallucinate" missing geometry, turning incomplete, open point clouds into fully closed, watertight room structures.
Higher-Level Structure: Lines, Graphs, and Programs
While planes and right angles capture the macro-geometry of rooms, man-made objects and buildings also exhibit higher-level structural regularities like symmetry, repetition, and part alignment.
Early work by Joachim Bauer in Towards Feature-Based Building Reconstruction from Images recognized that tracking 3D line segments and vanishing points provides a much more stable foundation for architectural reconstruction than tracking points alone, allowing systems to infer straight edges and parallel facades even through heavy image noise. To capture more complex architectural profiles, Changchang Wu et al. introduced Schematic surface reconstruction, which abstracts buildings as a network of swept surfaces (combining 2D floor plans with vertical profiles). This structural prior allows the system to compress data massively while filling in large occluded regions. In large-scale aerial mapping, Julian Straub et al. combined multi-modal sensor data with a structured geometric prior in Semantically-Aware Aerial Reconstruction from Multi-modal Data, demonstrating that enforcing architectural parallelism and orthogonality allows a model to successfully distinguish actual building boundaries from sensor artifacts and vegetation.
For object-level reconstruction, shapes can be regularized by treating them as assemblies of geometric primitives. Qian He et al. developed Single Image 3D Object Estimation with Primitive Graph Networks, which predicts 3D shapes as a graph of bounding boxes. By performing global reasoning over the graph, the network explicitly preserves symmetry and part alignments that are routinely lost in unstructured voxel or point outputs.
Taking this concept to its logical conclusion, Yonglong Tian et al. bypassed traditional spatial grids entirely in Learning to Infer and Execute 3D Shape Programs. They developed a neuro-symbolic model that expresses 3D shapes as executable computer programs containing loops, reflections, and subroutines. Because the output format itself is a program, regularities like the symmetry of a chair's legs or the repetitive spacing of a building's windows are not merely encouraged by a loss function—they are mathematically guaranteed by the execution of the code.
Unstructured Output (Voxels/Points): [Model] ──> [Independent Spatial Coordinates] ──> (Asymmetric Noise)
Neuro-Symbolic Output (Program): [Model] ──> `for i in range(4): emit_leg()` ──> (Perfect Symmetry)
Why Pure Data-Driven Scaling Falls Short
A natural counterargument in the era of foundation models is that explicit geometric priors are unnecessary; given enough training data, a sufficiently large neural network should implicitly learn the laws of geometry.
However, empirical evidence suggests otherwise. Mateusz Michalkiewicz et al. introduced a rigorous evaluation framework in GIQ: Benchmarking 3D Geometric Reasoning of Vision Foundation Models with Simulated and Real Polyhedra. By testing state-of-the-art vision foundation models on basic geometric solids (polyhedra), the authors discovered that even the most advanced models frequently fail to maintain fundamental geometric regularities like planarity, symmetry, and perpendicularity. Instead of understanding the underlying mathematical principles of a shape, these models rely on statistical pattern matching, leading to physically impossible distortions and asymmetric artifacts on even simple objects like cubes.
This finding underscores the enduring relevance of this research direction: geometric regularity is not a superficial pattern that can be reliably learned through passive observation. It is a fundamental structural constraint. Explicitly embedding these geometric priors into our representations ensures that 3D reconstructions remain accurate, efficient, and physically plausible.
Key Foundational Work
Towards Feature-Based Building Reconstruction from Images (2003) — contributed 1 concept elements to this direction.
Reconstruction of structured scenes from two uncalibrated images (2005) — contributed 1 concept elements to this direction.