TL;DR
This paper introduces 3D-STD, a novel method for generating realistic 3D face aging results by disentangling facial shape and texture. It leverages 3D reconstruction and a novel Empirical Mode Decomposition (EMD)-based texture generation to achieve superior aging accuracy and identity preservation, particularly for high-frequency aging details.
What Problem Does This Paper Address?
Generating realistic facial images that depict an individual aging across their entire lifespan is a challenging task in computer vision. Existing methods often struggle to accurately model both the significant shape deformations that occur during youth and the subtle texture changes that persist throughout life. Many approaches either focus solely on texture or use entangled shape-texture representations, leading to suboptimal results. A key challenge is capturing the high-frequency details of aging, such as wrinkles and furrows, which are often lost due to the spectral bias of generative networks. Furthermore, producing plausible 3D face aging results, which allow for viewing from multiple angles, is rarely accomplished.
Methods
The core of the proposed method, 3D-STD, lies in its ability to disentangle shape and texture information in a 3D space.
- 3D-Aware Shape-Texture Disentanglement: The input 2D face image is first processed using a 3D reconstruction technique (PRNet) to obtain a UV position map and a texture map. These represent the 3D shape and texture, respectively. This canonical, pose-normalized representation in UV space is believed to facilitate learning complex aging transformations.
- Dual-Branch Aging Network: The disentangled shape and texture representations are then fed into separate aging branches.
- The shape branch uses a shape encoder and decoder, conditioned on a latent age code, to generate an aged UV position map.
- The texture branch employs an identity encoder to extract identity features from the original texture. Crucially, it uses a novel EMD-based texture generation method to produce the aged texture map.
- EMD-based Texture Generation: To address the spectral bias issue and capture high-frequency aging details, Empirical Mode Decomposition (EMD) is used. EMD decomposes the texture into several Intrinsic Mode Functions (IMFs) representing different frequency ranges and a residue. The network is trained to generate these IMFs and the residue, which are then combined to form the final aged texture. This approach helps the network overcome limitations in modeling high-frequency content.
- Renderer: Finally, a renderer is used to synthesize the output 3D face aging result from the aged shape and texture representations.
Datasets
- FFHQ-Aging dataset: This dataset, pruned to 28,701 images and divided into 6 age groups ({0-2, 3-6, 7-9, 15-19, 30-39, 50-69}), was used for training and evaluation.
Key Insights and Ideas
- 3D Disentanglement is Key: Explicitly disentangling shape and texture in a 3D space, rather than relying on 2D entangled representations, leads to superior aging results, especially for significant shape changes in younger age groups.
- EMD for High-Frequency Details: The spectral bias of generative models hinders the capture of fine aging details. EMD provides a data-adaptive way to decompose signals into frequency components, allowing the network to explicitly learn and generate these crucial high-frequency aging features (wrinkles, furrows).
- Shape vs. Texture Aging Dynamics: The paper highlights that shape deformation is more prominent in younger stages, while texture transformation is continuous. Their dual-branch approach effectively models these distinct dynamics.
- Trade-off in Identity Preservation: Achieving radical age transformations can sometimes conflict with perfect identity preservation, as evaluated by automated systems. However, the authors argue that their method produces more natural and realistic aging, even if automated identity metrics show a slight dip.
Key Results
- State-of-the-Art Performance: The proposed 3D-STD method achieves state-of-the-art performance in terms of aging accuracy, outperforming methods like LATS, DLFS, and CUSP.
- Superior Aging Accuracy: Quantitative results show significant improvements in aging accuracy, particularly in younger age groups (where shape changes are drastic) and older age groups (where high-frequency texture details are crucial). For example, the overall aging accuracy is improved.
- Plausible 3D Aging: The method successfully generates plausible 3D face aging results, allowing for multi-view synthesis, a capability rarely seen in prior work.
- Identity Preservation: While DLFS shows slightly better identity preservation scores, 3D-STD maintains a competitive level, especially considering the significant age transformations.
- Qualitative Improvements: Visual comparisons demonstrate more realistic shape deformations and finer texture details, including improved rendering of accessories like glasses.
Limitations and Future Work
- Identity Preservation Trade-off: The authors acknowledge that radical shape transformations might lead to a perceived decrease in identity preservation by automated systems, suggesting a potential conflict between extreme aging and identity metrics.
- Future Work: The paper suggests that their work advances the field of 3D face aging, an area with significant research value and potential for further exploration. The method's ability to produce 3D models opens avenues for more advanced 3D face manipulation tasks.
Summary updated