TL;DR
This paper introduces FOUND, a novel method for reconstructing 3D human feet from a small number of images. It leverages a new synthetic dataset (SynFoot) and an uncertainty-aware normal prediction network to achieve robust and accurate reconstructions, outperforming traditional photogrammetry pipelines, especially in few-view scenarios.
What Problem Does This Paper Address?
Accurate 3D reconstruction of human body parts, particularly the foot, is crucial for industries like fashion, orthotics, and health monitoring. Existing methods face significant challenges: expensive scanning equipment is inaccessible to most, phone-based sensors often yield noisy point clouds, and traditional Structure from Motion (SfM) and Multi-View Stereo (MVS) pipelines require a large number of overlapping images for high-quality results. Furthermore, the lack of large-scale, paired image and 3D ground truth datasets for feet has hindered the development of robust deep learning approaches. This paper aims to bridge this gap by enabling accurate foot reconstruction from a minimal number of images, even those captured by consumer devices.
Methods
The FOUND approach is built upon four key contributions:
- SynFoot Dataset: A large-scale synthetic dataset of 50,000 photorealistic foot images generated using Blender. This dataset is paired with ground truth surface normals, keypoints, and silhouettes, providing a rich resource for training.
- Uncertainty-Aware Normal Prediction: They train a deep neural network to predict per-pixel surface normals along with their uncertainty. This network is trained on the synthetic SynFoot data and then fine-tuned using aggressive data augmentation to generalize to real-world images. The uncertainty estimation is crucial for down-weighting unreliable predictions.
- Generative Foot Model Optimization: They utilize the FIND (Foot Implicit Network Deformation) model, a generative model for human feet. This model is optimized using predicted surface normals and keypoints from real images. The optimization process leverages differentiable rendering to align the generative model with the input views.
- Uncertainty-Weighted Optimization: The optimization scheme incorporates the predicted uncertainties from the normal predictor. This allows the system to be more robust to noisy or inaccurate normal predictions, particularly in challenging areas or for occluded keypoints.
Datasets
- SynFoot: A newly introduced synthetic dataset containing 50,000 photorealistic foot images. It includes ground truth labels for silhouettes, surface normals, and keypoints. This dataset was generated using Blender and is designed to capture variations in foot appearance and pose.
- Evaluative Dataset: A smaller dataset comprising 474 images of 14 real feet. These images are paired with ground truth per-pixel surface normals and high-resolution 3D scans. Camera extrinsics and intrinsics for this dataset were obtained using Structure from Motion (SfM).
Key Insights and Ideas
- Synthetic Data for Generalization: The paper demonstrates that a synthetic dataset, even with a limited number of real-world scans used for generation, can be sufficient to train a normal predictor that generalizes well to real-world images, provided aggressive data augmentation is used.
- Uncertainty is Key: Explicitly predicting and utilizing uncertainty in surface normal estimation is shown to be critical for robust reconstruction, especially in few-view settings and for handling noisy predictions.
- Few-View Reconstruction is Achievable: The combination of a strong generative prior (FIND model) and uncertainty-aware normal prediction allows for accurate foot reconstruction with as few as 3-5 input views, a significant improvement over traditional methods.
- Bridging the Domain Gap: Aggressive data augmentation techniques are vital for closing the gap between synthetic training data and real-world observations.
Key Results
- Normal Prediction: The proposed uncertainty-aware normal predictor outperforms baseline methods (COLMAP and Bae et al. [10]) on their evaluative dataset, achieving a mean normal error of 11.30° compared to 28.24° for COLMAP and 25.50° for the augmented baseline.
- 3D Reconstruction: The FOUND method significantly outperforms COLMAP in reconstruction quality for few-view scenarios. While COLMAP requires 30+ views for comparable accuracy, FOUND maintains high accuracy with as few as 3-5 views.
- Quantitative Metrics: In the many-view case, FOUND achieves a lower NN chamfer error (15.2 µm vs. 16.1 µm for COLMAP) and a better NN normal error (12.9° vs. 20.3° for COLMAP).
- Qualitative Results: Visualizations show that FOUND produces smoother reconstructions with better surface normal consistency compared to COLMAP, especially around complex areas like the toes.
Limitations and Future Work
- Constrained Poses: The current evaluation focuses on feet placed on a floor. The authors acknowledge that the method is not inherently restricted to this but would require extension for posed feet in arbitrary positions.
- Generalization to Other Body Parts: The authors suggest extending the method to other body parts, such as hands, as a future direction.
- AR Tracking Integration: While the method can work with camera parameters obtained from AR tracking, the current evaluation relies on SfM-derived camera poses. Further qualitative evaluation with AR-tracked cameras is presented.
- Ground Truth Data: The creation of the evaluative dataset with ground truth surface normals was labor-intensive, highlighting the ongoing challenge of obtaining such data.
Summary updated