Tree-Structured Shading Decomposition

Chen Geng, Hong-Xing Yu, Sharon Zhang, Maneesh Agrawala, Jiajun Wu

Inverse rendering aims to recover 3D properties—lighting, material, and geometry—from a single 2D image. While traditional methods using physical models like BRDFs are accurate, they often function as "black boxes," making it difficult for users to intuitively edit specific material properties. Researchers from Stanford University have introduced a novel approach that decomposes shading into an interpretable, hierarchical "shade tree," enabling modular editing and relighting.

The Shade Tree Framework

The core innovation is the revival of the "Shade Tree" concept from the 1980s, modernized through a two-stage hybrid pipeline. The system uses a Domain-Specific Language (DSL) where shading is represented as a tree structure. Leaf nodes represent basic elements like albedo, highlights, and diffuse reflection, while interior nodes define operations such as multiply, screen, and mix.

Overview of the tree-structured shading decomposition pipeline, showing reconstruction, reuse, and editing.

The decomposition process involves two primary stages:

  1. Recursive Amortized Inference: An auto-regressive model (built on VQ-VAE and PixelSNAIL) predicts the initial tree structure and parameters by analyzing the input image layer-by-layer. It effectively "guesses" the modular composition of the shading.
  2. Optimization-based Finetuning: To ensure high fidelity, a search algorithm (BasinCMA) refines the tree's continuous parameters, such as highlight sharpness or albedo intensity, to perfectly match the original image.

Key Insights and Results

By combining deep learning for structural inference with optimization for precision, the model achieves superior performance over standard CNN, LSTM, and Transformer baselines. On the "Realistic" synthetic dataset, the model achieved a PSNR of 30.89, significantly outperforming the CNN baseline of 26.50.

A major advantage of this representation is interpretability. Users can perform "shading analogies"—transferring the highlight structure of one object to another by simply swapping nodes. In analogy tests, the model achieved a PSNR of 32.17, while baselines struggled around 5.0, indicating a deep structural understanding of shading rather than simple pixel memorization.

Applications of the decomposed representation including vectorization and object relighting.

Applications and Limitations

The decomposed shade trees allow for advanced downstream tasks, including vectorization and relighting of "in-the-wild" images. For instance, a metallic sculpture or a simple mug can be converted into a modular asset where lighting can be adjusted by swapping environment maps or tweaking specific highlight branches in the tree.

Currently, the system relies on "MatCaps" or spherical reflectance maps as input. Future work aims to extend this to spatially varying materials—where different parts of an object have different properties—and integrate these tools more deeply into professional vector illustration workflows.

Summary updated

Abstract

We study inferring a tree-structured representation from a single image for object shading. Prior work typically uses the parametric or measured representation to model shading, which is neither interpretable nor easily editable. We propose using the shade tree representation, which combines basic shading nodes and compositing methods to factorize object surface shading. The shade tree representation enables novice users who are unfamiliar with the physical shading process to edit object shading in an efficient and intuitive manner. A main challenge in inferring the shade tree is that the inference problem involves both the discrete tree structure and the continuous parameters of the tree nodes. We propose a hybrid approach to address this issue. We introduce an auto-regressive inference model to generate a rough estimation of the tree structure and node parameters, and then we fine-tune the inferred shade tree through an optimization algorithm. We show experiments on synthetic images, captured reflectance, real images, and non-realistic vector drawings, allowing downstream applications such as material editing, vectorized shading, and relighting. Project website: https://chen-geng.com/inv-shade-trees
BibTeX
@inproceedings{geng2023tree,
  author = {Chen Geng and Hong-Xing Yu and Sharon Zhang and Maneesh Agrawala and Jiajun Wu},
  title = {{Tree-Structured Shading Decomposition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure