TL;DR
This paper introduces the first method for creating personalized 3D dental models and capturing their motion using just a single smartphone camera. By combining mesh and signed distance function (SDF) representations, their approach enables realistic facial avatars with accurate teeth that move naturally with expressions, overcoming previous limitations in low-cost, accessible facial capture.
What Problem Does This Paper Address?
Creating realistic 3D digital humans, especially for virtual and augmented reality, is a major goal. While significant progress has been made in capturing facial skin and expressions, the teeth have largely been overlooked. Existing methods often omit teeth, leading to an "uncanny valley" effect, or require expensive, multi-camera setups that aren't accessible to most users. The complex interplay between teeth and lips, especially during occlusion (when one is hidden by the other), presents a significant challenge for accurate performance capture. Furthermore, the unique shapes and alignments of individual teeth are crucial for identity representation, making their accurate reconstruction and animation important for truly personalized avatars.
Methods
The core of this work lies in a novel "analysis-by-synthesis" approach that reconstructs and tracks teeth using a single phone camera. Key innovations include:
- Representation Adaptation: They use a dual representation for teeth: a mesh for efficient rendering and animation, and a neural Signed Distance Function (SDF) for robust collision detection and preventing teeth-lips interpenetration. This allows them to leverage the strengths of both representations.
- Teeth-Assembled Facial Rig: They extend the FLAME facial model to incorporate the teeth. This involves defining rigid transformation parameters for the teeth and using blend weights to attach them to the skull and jaw joints, ensuring they move realistically with the rest of the face.
- Semantic Mask Supervision: To overcome the challenge of similar-looking teeth, they employ a foundation model for semantic teeth segmentation. This provides per-tooth masks that are used to supervise the optimization process, ensuring precise alignment.
- SemanticXY Supervision: To address issues with initial pose estimation and prevent convergence to local minima, they introduce a semantic loss that uses optimal transport to establish long-range correspondences between rendered and target masks. This helps guide the optimization process more effectively.
- Occlusion-Aware Optimization Strategies: They tackle the difficult problem of teeth occlusion by the lips. They propose two specialized strategies: a "face-vanishing" strategy to help teeth emerge from occlusion and a "teeth-hiding" strategy to ensure teeth are correctly occluded by facial features.
Datasets
The authors captured their own RGB videos using an iPhone 14 from four participants. This dataset was collected in two phases:
- Phase 1: Participants maintained a "toothy expression" for about 20 seconds while the phone moved in an S-shaped trajectory.
- Phase 2: The camera was positioned directly in front of the participants as they performed various expressions involving dynamic tooth movements.
They also created a synthetic dataset using Unreal MetaHuman to replicate their capture setting, which includes ground truth depth maps for quantitative evaluation.
Key Insights and Ideas
- Democratizing High-Quality Avatars: The most significant insight is that high-quality, personalized facial avatars with accurate teeth can be achieved with a single, common device like a smartphone, making this technology much more accessible.
- The Power of Dual Representations: Combining mesh and SDF representations for teeth is crucial. The mesh allows for efficient rendering and integration into existing facial animation pipelines, while the SDF is essential for enforcing physically plausible interactions (like preventing interpenetration) which are vital for realistic animation.
- Leveraging Semantic Information is Key: The visual similarity of teeth makes traditional RGB-based losses insufficient. Using semantic masks for individual teeth provides a strong supervisory signal that significantly improves alignment and reconstruction accuracy.
- Occlusion is a Major Hurdle, But Solvable with Priors: The proposed occlusion-aware strategies highlight that by understanding the specific anatomical constraints (e.g., teeth are occluded by lips), one can design effective optimization techniques to handle challenging visibility transitions.
Key Results
- Performance Capture: The method demonstrates superior performance capture compared to state-of-the-art methods like VHAP and MonoNPHM, producing more accurate teeth motion and geometry that aligns well with the target. Quantitative results show a significant improvement in Chamfer Distance (CD) for performance capture.
- Shape Reconstruction: The method also excels at reconstructing detailed teeth shapes, outperforming methods like TensoSDF, particularly in capturing fine geometric details.
- Robustness: Ablation studies show that the SemanticXY loss significantly reduces sensitivity to initial pose, leading to more robust convergence. The contact loss effectively prevents artifacts caused by teeth-face intersection. The occlusion-aware strategies are shown to prevent the tracking from getting stuck in local minima during challenging expressions.
Limitations and Future Work
- Segmentation Dependency: The method relies on accurate teeth and face segmentation. While they use the semi-automatic SAM2, further automation of this process would improve usability. They suggest that advances in class-specific segmentation models could benefit their approach.
- Face Reconstruction Orthogonality: The face reconstruction accuracy is an orthogonal direction, implying that their teeth tracking method could be applied to other facial parametric models.
- Future Directions: The authors suggest that improving face parametric models could further enhance their method. They also imply that their teeth tracking could be integrated with other facial components or used for a wider range of applications.
Summary updated