Direct Variational Perspective Shape from Shading with Cartesian Depth Parametrisation

Yong Chul Ju, Daniel Maurer, Michael Breuß, Andres BruhnPerspectives in Shape Analysis 2016 · DOI · DBLP

Abstract

Shape from Shading (SfS) is a fundamental problem in computer vision aimed at recovering 3D geometry from a single intensity image. This paper introduces a robust variational framework for perspective SfS that operates directly on Cartesian depth. By combining a physics-based data term with an edge-preserving second-order regularizer and an efficient alternating numerical solver, the proposed method achieves high-quality reconstructions. The approach is inherently integrable and demonstrates significant resilience to noise and data gaps compared to traditional PDE-based methods.

Introduction

Reconstructing a 3D world from a 2D image remains a central challenge in computer vision. Shape from Shading (SfS) attempts this by analyzing variations in brightness. However, traditional SfS methods often rely on simplifying assumptions, such as orthographic projection, which fail in close-up scenarios like medical endoscopy. Furthermore, many state-of-the-art solvers treat SfS as a Partial Differential Equation (PDE) problem. While efficient, these solvers are often "brittle"; noise or missing data can cause the reconstruction to fail due to a lack of global smoothness constraints.

Existing variational (energy-minimization) methods have attempted to address these issues but often utilize radial depth (distance from the camera center), which is geometrically less intuitive, or require complex integrability constraints to ensure the resulting surface normals form a valid 3D shape.

Methodology

The authors propose a novel framework characterized by three key innovations:

Direct Cartesian Parametrization

Unlike previous models using radial depth, this approach formulates the entire model using Cartesian depth ($z$). This choice simplifies integration with other computer vision tasks, such as stereo vision, and ensures that the reconstructed surface is inherently integrable.

Perspective projection geometry with Cartesian depth parametrization.

As shown in the figure above, the surface $\mathcal{S}$ is defined by the Cartesian depth $z(\mathbf{x})$ at each image point $\mathbf{x}$. The model explicitly accounts for the $1/r^2$ light attenuation law, which helps resolve the classic convex-concave ambiguity in SfS.

Edge-Preserving Regularization

To handle noise without losing detail, the framework employs a second-order regularizer based on the Frobenius norm of the Hessian. A Charbonnier function is applied to prevent oversmoothing of sharp features, allowing the algorithm to maintain distinct edges while smoothing flat regions.

Alternating Explicit Scheme

To solve the resulting energy functional efficiently, the authors utilize a coarse-to-fine approach. They introduce an "alternating" numerical solver that ignores certain complex gradient terms in alternating iterations. This allows for significantly larger time steps, resulting in speedups of up to four orders of magnitude compared to standard explicit schemes.

Experimental Results

The model was evaluated on several synthetic datasets (Sombrero, Suzanne, Stanford Bunny, and Dragon) and real-world facial images.

  • Accuracy and Noise Robustness: In synthetic tests, the model achieved low Relative Surface Error (RSE). When subjected to Gaussian noise ($\sigma = 20$), the variational method maintained errors between 1-5%, whereas traditional PDE methods saw errors spike to 11-20%.
  • Inpainting Capabilities: The global smoothness term allows the model to "fill in" missing data. If portions of the image are obscured, the regularizer inpaints the geometry based on the surrounding context.
  • Performance: The alternating explicit scheme made the reconstruction of high-resolution images computationally feasible.

Conclusion

This research presents a significant advancement in perspective Shape from Shading by prioritizing Cartesian depth and robust variational regularization. While currently limited by the Lambertian surface assumption (ignoring specular highlights), the framework provides a solid foundation for future work. The authors suggest extending this "Cartesian-first" approach to multi-view scenarios, potentially merging SfS with stereo vision for enhanced 3D mapping accuracy.

Summary updated

BibTeX
@inproceedings{ju2016direct,
  author = {Yong Chul Ju and Daniel Maurer and Michael Breuß and Andres Bruhn},
  title = {{Direct Variational Perspective Shape from Shading with Cartesian Depth Parametrisation}},
  booktitle = {Perspectives in Shape Analysis},
  year = {2016},
  doi = {10.1007/978-3-319-24726-7\_3},
}
Figure