Shadow Art Revisited: A Differentiable Rendering Based Approach

Kaustubh Sadekar, Ashish Tiwari, Shanmuganathan RamanWACV 2022

TL;DR

This paper introduces a novel framework for generating 3D sculptures that cast specific artistic shadows, using differentiable rendering to optimize the 3D geometry from desired 2D shadow images. This approach bypasses the need for explicit 3D ground truth, enabling the creation of complex and abstract shadow art.

What Problem Does This Paper Address?

Creating artistic shadow sculptures, where 3D objects cast interesting 2D shadows, has traditionally been a manual and trial-and-error process. While prior work has explored this problem using optimization, it often required significant manual intervention or deformation of input images. This paper addresses the challenge of automatically generating 3D sculptures that produce desired shadow effects from a set of target shadow images and their corresponding projection information, without needing explicit 3D models or altering the input shadows.

Methods

The core of the proposed method is a differentiable rendering-based optimization framework. The authors explore two primary 3D representations: voxels and meshes.

  • Voxel-based Optimization: The 3D object is represented as a grid of voxels, where each voxel stores density information. A differentiable volumetric renderer is used to generate shadow images from this voxel grid. The optimization process minimizes the difference between the rendered and target shadow images, along with regularization terms to ensure smoothness and prevent extreme deformations.
  • Mesh-based Optimization: The 3D object is represented as a mesh. The optimization starts with a base mesh (an icosphere) and learns per-vertex displacements to deform it into the final sculpture. A mesh-based differentiable silhouette renderer is employed. The loss function includes image-based loss, normal consistency, Laplacian smoothing, and edge length regularization to achieve a smooth and well-formed mesh that casts the desired shadows.

The key novelty lies in using differentiable rendering to directly optimize the 3D geometry from 2D shadow inputs, enabling gradient flow from image space to 3D shape parameters.

Datasets

Not discussed in the paper. The paper uses custom-generated shadow images and sketches as input for the optimization process.

Key Insights and Ideas

  • Differentiable Rendering for Artistic Creation: The paper demonstrates that differentiable rendering, typically used for reconstruction, can be effectively repurposed for creative artistic applications like shadow art.
  • Bypassing 3D Ground Truth: The ability to learn 3D geometry solely from 2D shadow images and projection information is a significant advantage, as it removes the dependency on expensive and often unavailable 3D datasets.
  • Voxel vs. Mesh Trade-offs: Voxel-based methods are better at handling sharp discontinuities and holes in shadows but can be memory-intensive and produce discretized results. Mesh-based methods yield smoother results but may struggle with sharp features due to limitations of the base mesh (e.g., icosphere).
  • Importance of Regularization: The study highlights that image-based loss alone is insufficient. Regularization terms (edge length, Laplacian, normal consistency) are crucial for producing plausible and smooth 3D sculptures that avoid distortions.
  • Beyond Simple Objects: The framework successfully generates shadows for complex inputs like faces and animated characters, showcasing its versatility.

Key Results

  • The proposed framework successfully generates 3D sculptures that cast desired shadows from various viewpoints, outperforming prior methods in handling shadow inconsistencies without deforming input images.
  • Qualitative results show plausible 3D sculptures and their corresponding shadows for diverse inputs, including abstract shapes, faces, and characters.
  • Quantitative evaluation using IoU and Dice Score demonstrates comparable performance between voxel and mesh-based approaches in shadow rendering accuracy.
  • The mesh-based approach generally achieves higher normal consistency, indicating smoother 3D sculptures.
  • The paper demonstrates the framework's applicability to sketch-based 3D reconstruction, successfully creating 3D models of everyday objects from hand-drawn sketches.
  • 3D printed models of some generated sculptures are presented, validating their physical realizability.

Limitations and Future Work

  • Printing Complex Structures: Facial structures and certain complex shapes generated by the voxel-based method are difficult to 3D print due to missing connectivity. Converting these to meshes for printing remains a challenge.
  • Mesh Limitations: The mesh-based approach can exhibit discontinuities near non-convex regions, partly due to the limitations of the base mesh (icosphere) in handling sharp features, especially with regularization.
  • Static Nature: The current work focuses on static sculptures. The authors suggest exploring dynamic shadow art as a future direction.
  • Further Exploration of Representations: While voxel and mesh are discussed, other representations could be explored.
  • Improving Mesh Generation: Further research into converting voxel grids to printable meshes and handling sharp discontinuities in mesh-based methods is suggested.

Summary updated

BibTeX
@inproceedings{sadekar2022shadow,
  author = {Kaustubh Sadekar and Ashish Tiwari and Shanmuganathan Raman},
  title = {{Shadow Art Revisited: A Differentiable Rendering Based Approach}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure