Abstract
Reconstructing complete 3D facial geometry from a single, potentially noisy or occluded depth view is a significant challenge in computer vision. We present the Attention Guided Generative Adversarial Network (AGGAN), a framework that leverages voxel-based representations and dual attention mechanisms to recover high-resolution 3D faces. By integrating spatial and channel-wise attention, AGGAN effectively filters noise and prioritizes salient features, enabling robust reconstruction even under extreme head poses and sensor noise. Experimental results demonstrate that AGGAN outperforms traditional surface reconstruction methods, achieving high accuracy and anatomical plausibility.
Introduction
Reconstructing a 3D face from a single depth view—such as data captured by consumer-grade depth sensors—is a classic ill-posed problem. When a subject is captured from a side profile, nearly half of the facial geometry is occluded and technically unknown. Traditional approaches generally fall into two categories: surface reconstruction (e.g., Screened Poisson Reconstruction), which often produces holes or distortions in missing regions, and non-rigid registration, which warps generic templates but struggles with extreme poses and requires manual landmarking.
To address these limitations, we propose AGGAN, a fully automated, data-driven system designed to "fill in the blanks" for occluded facial regions while remaining robust to sensor noise and extreme head orientations.
Methodology
AGGAN Architecture
The core of our approach is a conditional Generative Adversarial Network (cGAN) that operates on a $128 \times 128 \times 128$ voxel grid. Unlike 2D-to-3D methods that produce meshes directly, our model assigns a probability to each voxel in the grid, representing its likelihood of being part of the facial surface.
Attention Mechanisms
The "secret sauce" of the AGGAN generator lies in its dual attention modules:
- Spatial Attention: This module enables the model to focus on the facial region specifically, effectively ignoring noisy background pixels in the input depth map.
- Channel-wise Attention: This mechanism weights different feature layers based on their utility for predicting 3D structure, ensuring that the most informative features are prioritized during reconstruction.
Loss Functions and Training
Training is guided by a multi-faceted loss function. Beyond the standard Adversarial Loss for realism, we employ a weighted binary cross-entropy loss to manage the sparsity of the 3D grid (where most voxels are empty space) and an L1 sparsity constraint to ensure the resulting facial surface is sharp and well-defined.
Data Synthesis and Training
Due to the scarcity of large-scale, high-quality 3D facial scans, we utilized data synthesis. We leveraged the 300W-LP dataset to generate over 75,000 training samples. To ensure real-world robustness, we synthetically augmented the data with random Gaussian noise and rotated faces up to 90-degree yaw, simulating extreme poses and sensor imperfections.
Results and Discussion
Quantitative Performance
AGGAN achieved a Mean Intersection-over-Union (IoU) of approximately 0.99 across multiple test sets (AFW, IBUG, and LFPW). This indicates near-perfect overlap with ground truth 3D shapes, significantly outperforming traditional methods like Non-rigid ICP (NICP) and Screened Poisson Reconstruction (SPSR), which typically degenerate at poses beyond 45 degrees.
Qualitative Robustness
As shown in the qualitative results, AGGAN maintains a stable and complete facial structure even under challenging conditions such as extreme head poses, various facial expressions, and significant sensor noise.
Conclusion and Future Work
AGGAN demonstrates that learning a statistical "prior" of human facial anatomy allows for the successful hallucination of missing geometry from single-view depth data. While the $128^3$ resolution provides high fidelity, future work will explore even denser voxel grids or residual learning approaches to capture finer details like eyelid creases. Additionally, we aim to further bridge the gap between synthetic and real-world data by training on more diverse, captured depth datasets.
Summary updated