TL;DR
This paper introduces a novel, differentiable loss function called Geometric Rényi Divergence (GRD) for estimating 3D shape from a single 2D semantic segmentation. GRD offers a robust and wide-ranging optimization landscape, outperforming existing methods in both direct 3D model fitting and training CNNs for shape regression.
What Problem Does This Paper Address?
Estimating the 3D shape of an object from a single 2D image is a challenging problem. While silhouettes have been used, semantic segmentation (identifying object parts) offers richer information. However, directly using pixel-wise semantic segmentation for 3D shape estimation is difficult because the correspondence between a 2D pixel and a 3D model vertex is often one-to-many and the segmentation itself is discrete, leading to discontinuous loss functions that are hard to optimize. Existing methods using differentiable renderers often struggle with precise alignment or suffer from issues like model shrinking. This paper aims to bridge this gap by proposing a robust, differentiable loss that leverages semantic segmentation for accurate 3D shape estimation, even from challenging initial conditions.
Methods
The core contribution is the Geometric Rényi Divergence (GRD) loss function. Here's how it works:
- Probabilistic Representation: Both the input semantic segmentation (pixel labels) and the projected 3D model vertices are represented as Mixtures of Gaussians (MoGs). This "softens" the discrete labels into continuous probability distributions, making them amenable to gradient-based optimization.
- Geometric Rényi Divergence (GRD): Instead of standard divergences, the authors propose GRD, a variant of Rényi divergence. GRD is designed to be non-negative and have a wide basin of convergence, meaning it can handle large initial errors in pose and shape. The paper derives a closed-form solution for GRD when comparing two MoGs, specifically for the second-order Rényi entropy (q=2).
- Analysis-by-Synthesis: The GRD loss can be directly used to optimize 3D model parameters (shape and pose) by minimizing the discrepancy between the projected semantic labels and the input pixel labels. This is done by rendering the 3D model and comparing its projected semantic segmentation to the ground truth.
- CNN-based Regression: The GRD loss is also integrated as a supervisory signal for training a CNN that directly regresses 3DMM parameters from a single image.
- Automatic Vertex Labeling: To apply the GRD loss, semantic labels need to be assigned to 3D model vertices. The authors propose an automatic method involving pre-training a face parsing network, rendering the 3D model, and then using a majority vote to assign labels to vertices.
Datasets
- CelebAMask-HQ: Used for training the face parsing network and for training the CNN-based regression model.
- AFLW [20] and AFLW2000-3D [33]: Used for evaluating the performance of landmark detection.
Key Insights and Ideas
- Softening Discrete Labels is Key: Representing both pixel and vertex labels as MoGs is crucial for enabling differentiable optimization. This transforms a hard, discontinuous problem into a smooth one.
- GRD's Robustness: The geometric Rényi divergence is shown to be more robust than other metrics (like L2 or IoU) for alignment tasks, especially when dealing with large initial errors or scale changes. Its loss landscape is more favorable for optimization.
- Semantic Segmentation as a Richer Cue: The paper demonstrates that fine-grained semantic segmentation provides a powerful signal for 3D shape estimation, surpassing the limitations of simpler cues like silhouettes or sparse landmarks.
- Bridging Analysis-by-Synthesis and CNNs: The GRD loss is versatile, serving effectively in both traditional optimization frameworks and modern deep learning pipelines.
Key Results
- Direct Optimization: GRD-based direct optimization shows superior convergence to the ground truth compared to NMR and SoftRas, especially in challenging cases with large pose variations. Table 1 shows that GRD achieves a mean IoU of 0.931, significantly outperforming NMR (0.789) and SoftRas (0.423).
- CNN-based Regression: The CNN trained with GRD loss achieves comparable results to state-of-the-art landmark-based methods for modest pose angles, as shown in Table 2. Qualitative results (Figure 9) demonstrate successful reconstruction of 3D faces, including ears, under various transformations.
Limitations and Future Work
- Single Object Class: The current method is restricted to a single object class (faces) via the 3DMM.
- Automatic Labeling Consistency: While the automatic vertex labeling is effective, human annotators can have inconsistencies, and the method aims for aggregate optimality.
- Future Work: The authors suggest extending the approach to other computer vision tasks like point cloud registration, image registration, and general 3D reconstruction. They also propose its use for multiview silhouette fitting, extension to other object classes, and integration into self-supervised pipelines.
Summary updated