Reconstructing 3D Surfaces via Local Global Optimization

18 concepts13 papers1 proposal

Inferring a continuous 3D surface from 2D images, gradients, or sparse point clouds is a fundamentally under-constrained problem. If we attempt to solve for an entire surface globally all at once, the optimization landscape is often plagued by non-linearities, noise, and severe ambiguities—such as those found in classic shape-from-shading or shape-from-texture problems. Conversely, if we look only at a single point, the local data is insufficient to determine orientation or depth uniquely.

To resolve this tension, a powerful research direction has emerged around a two-phase, alternating local-global optimization framework. Instead of treating a surface as a single monolithic entity, these methods break the problem down into a "mosaic" of local patches, facets, or tiles. By alternating between optimizing these local elements independently to fit the observed data and blending them globally to ensure geometric continuity, algorithms can robustly reconstruct complex 3D shapes.

The Local Step: Exploiting Patch-Wise Uniqueness

Why break a surface into patches? At a macro scale, a surface can be incredibly complex, but at a micro scale, it can often be approximated by simple geometric primitives, such as planar facets or quadratic patches.

Xiong et al. demonstrated that while point-wise shape-from-shading is highly ambiguous, analyzing small image patches as quadratic surfaces often yields unique local shape solutions under known lighting. By treating local shapes as a distribution of likely proposals, the authors transformed a brittle global problem into a robust local one. Similarly, Badki et al. introduced "meshlets"—small, local mesh patches—as a category-agnostic prior, showing that a model can learn the manifold of natural local shapes to reconstruct geometry from sparse, noisy inputs.

In frameworks where the surface is represented as a piecewise linear mesh, such as the photometric stereo approach by Xie et al., each pixel corresponds to a polygonal facet. In the local step, target surface normals or vertex positions are calculated independently for each facet or patch to maximize data fidelity, completely ignoring whether the adjacent patches actually line up. This isolation allows the local step to capture fine-grained details, handle spatially-varying properties like albedo Sunkavalli et al., or perform view-specific geometric updates Ryu et al..

The Global Step: Stitching and Continuity

If left alone, the local step would produce a collection of disconnected, overlapping, or tearing tiles. The role of the global step is to reconcile these disparate local estimates into a single, continuous, and mathematically consistent surface.

This reconciliation is typically formulated as a global optimization problem—often a sparse linear system or a least-squares minimization—that enforces physical constraints like surface smoothness, continuity, and integrability. For instance, in the Surface-from-Gradients (SfG) pipeline presented by Xie et al. and also utilized by Shang et al., the global step takes the disconnected facets shaped by the local step and blends them into a continuous surface using least-squares optimization.

In modern neural or multi-view contexts, this global integration can take different forms. Ryu et al. use Poisson surface reconstruction to integrate local geometric updates from individual camera views into a seamless global 3D representation, preventing spatial drifting. No matter the specific mathematical tool, the global step acts as a regularizer, ensuring that local detail enhancements do not cause geometric inconsistencies or surface tearing Xie et al..

The Iterative Loop: Constraint Propagation

The true power of this approach lies in the iteration between the two steps. By repeatedly alternating between local data fitting and global smoothing, constraints are propagated across the entire surface, allowing local updates to inform global structure and vice versa.

This concept traces its roots back to early cooperative relaxation algorithms in computer vision. Woodham pioneered this by mapping image intensities to a 2D gradient space and iteratively narrowing down possible 3D orientations by enforcing local smoothness constraints among neighboring pixels. Similarly, Aliomonos et al. used an iterative propagation algorithm to resolve ambiguities in shape-from-texture, ensuring that local measurements converged to a globally consistent surface orientation.

┌────────────────────────────────────────────────────────┐
│                                                        │
│                  Initial Noisy Input                   │
│                                                        │
└───────────────────────────┬────────────────────────────┘
                            │
                            ▼
             ┌─────────────────────────────┐
             │         Local Step          │
             │  Optimize individual tiles/ │◄────────────┐
             │  patches for data fidelity  │             │
             └──────────────┬──────────────┘             │
                            │                            │
                            ▼                            │ Iterative
             ┌─────────────────────────────┐             │ Refinement
             │         Global Step         │             │   Loop
             │  Reconcile patches via      │             │
             │  least-squares / Poisson    │             │
             └──────────────┬──────────────┘             │
                            │                            │
                            └────────────────────────────┘
                            │
                            ▼
┌────────────────────────────────────────────────────────┐
│                                                        │
│             Continuous, Detailed 3D Surface            │
│                                                        │
└────────────────────────────────────────────────────────┘

In a modern alternating optimization loop, the global surface generated in iteration $t$ serves as a regularizing prior or a baseline for the local patch selection in iteration $t+1$. For example, Khan et al. apply this in surface remeshing by first optimizing individual patches in isolation and then performing a global smoothing pass with unlocked boundaries to eliminate artifacts at segment interfaces. In 3D scene inpainting, Chen et al. utilize a localized optimization phase guided by multi-view reprojection to seamlessly blend newly inserted 3D Gaussian patches into the surrounding global environment.

By decoupling high-frequency data fitting (the local step) from low-frequency structural continuity (the global step), these iterative mosaic frameworks turn highly non-linear, ill-posed reconstruction problems into a series of efficient, robust, and stable linear steps.

Key Foundational Work

A Cooperative Algorithm for Determining Surface Orientation from a Single View (1977) — contributed 2 concept elements to this direction.
Shape from Texture (1985) — contributed 1 concept elements to this direction.
A Cooperative Algorithm for Determining Surface Orientation from a Single View (1977) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_iterative_mosaic_reconstruction_9eb97e0f,
  title = {iterative mosaic reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-mosaic-reconstruction/rdr_9eb97e0f0abd4d3bb013c9b62e7b0771},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_iterative_mosaic_reconstruction_9eb97e0f_v1,
  title = {iterative mosaic reconstruction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-mosaic-reconstruction/rdr_9eb97e0f0abd4d3bb013c9b62e7b0771/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake