Abstract
Recent advances in single-image 3D face reconstruction have shown remarkable progress in various applications. Nevertheless, prevailing techniques tend to prioritize the global facial contour and expression, often neglecting the nuanced dynamics of the eye region. In response, we introduce an Eye Landmark Adjustment Module, complemented by a Local Dynamic Loss, designed to capture the dynamic features of the eyes area. Our module allows for flexible adjustment of landmarks, resulting in accurate recreation of various eye states. In this paper, we present a comprehensive evaluation of our approach, conducting extensive experiments on two datasets. The results underscore the superior performance of our approach, highlighting its significant contributions in addressing this particular challenge.
Introduction
Modern CNN-based 3D face reconstruction models, such as DECA, MICA, and Deep3DFaceRecon, excel at capturing global facial identity and broad expressions. However, they frequently fail to render fine-grained local dynamics, particularly in the eye region. Because the eyes occupy a relatively small portion of the face, standard loss functions and landmark detectors often overlook their specific states.
This leads to a phenomenon known as "eye state inconsistency": when fed an image of a person with closed eyes, these state-of-the-art models almost invariably generate a 3D mesh with eyes wide open. This limitation severely impacts the realism required for digital human animation, virtual makeup, and face reenactment.
Methodology
To address these challenges, the proposed framework integrates two novel components into a standard 3D Morphable Model (3DMM) regression pipeline: the Eye Landmark Adjustment Module (ELAM) and the Local Dynamic Loss (LDL).
Eye Landmark Adjustment Module (ELAM)
Standard facial landmark detectors often fail when eyes are closed. ELAM mitigates this by employing a dedicated eye-state detector (ResNet-50) that outputs a closure probability ($P$). This probability is used to mathematically adjust the y-axis positions of eyelid landmarks. For instance, a probability of 0.5 (indicating a half-closed eye) shifts the upper eyelid landmark halfway toward the lower eyelid, providing a flexible representation of various eye states rather than a binary open/closed classification.
Local Dynamic Loss (LDL)
The Local Dynamic Loss ensures the 3D mesh adheres to the corrected landmarks by calculating the relative distance between upper and lower eyelids (and lips) in both the 2D image and the projected 3D mesh. By heavily weighting these specific keypoint pairs, the loss function penalizes discrepancies in eye or mouth states, forcing the model to respect local dynamics without compromising global geometry.
Experiments and Datasets
The model was trained and evaluated using several key datasets:
- FFHQ: Approximately 15,000 high-quality face images were selected for general training.
- CEW (Closed Eyes In The Wild): A dataset of 2,000 images used to pretrain the ELAM detector and evaluate eye-state accuracy.
- NoW Benchmark: Used to measure overall 3D shape reconstruction error by comparing generated meshes against ground-truth 3D scans.
Results and Discussion
The integration of ELAM and LDL yields significant improvements in both local accuracy and global reconstruction quality.
Quantitative Performance
- Eye State Accuracy: On the CEW dataset, eye-state reconstruction accuracy skyrocketed from 51.7% to 96.6%, with the F1 score reaching 96.7%.
- Shape Reconstruction: On the NoW Benchmark, the full model achieved a median error of 1.336 mm, outperforming established baselines like PRNet (1.499 mm) and UMDFA (1.515 mm).
- Ablation Studies: Implementing LDL for both eyes and mouth reduced the median reconstruction error by 0.21 mm compared to a baseline without it, suggesting that local refinements actually contribute to better global alignment.
Conclusion
By prioritizing relative distances over absolute positions and utilizing probabilistic landmark adjustment, this approach successfully captures the nuanced motion of the eye region. While the current implementation relies on an external eye-state detector, future iterations could integrate this mechanism directly into the primary regression network for improved efficiency. Extending this methodology to other micro-expressions, such as nasal wrinkling or brow furrowing, remains a promising avenue for further enhancing 3D facial realism.
Summary updated