SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis

Anastasiia Kornilova, Marsel Faizullin, Konstantin Pakulev, Andrey Sadkov, Denis Kukushkin, Azat Akhmetyanov, Timur Akhtyamov, Hekmat Taherinejad, Gonzalo FerrerCVPR 2022 · DOI · arXiv

Introduction

High-fidelity 3D human digitization has traditionally been the domain of expensive multi-camera rigs and controlled laboratory environments. As mobile hardware evolves, there is a growing demand for "novel view synthesis" and 3D reconstruction directly from smartphone footage. However, consumer-grade sensors introduce significant noise, and the inherent "micro-motions" of human subjects—breathing, blinking, and slight postural shifts—violate the static-scene assumptions required by most state-of-the-art algorithms. The SmartPortraits dataset addresses these challenges by providing a large-scale, synchronized RGB-D dataset captured in realistic, handheld scenarios.

The SmartPortraits Dataset

The primary contribution of this work is a comprehensive dataset consisting of 1,000 video sequences featuring 200 unique individuals. Unlike existing datasets that focus on static objects or large-scale outdoor environments, SmartPortraits is specifically tailored for human portraits in "in the wild" conditions, such as offices, labs, and cafeterias.

Examples of the diverse subjects and indoor environments included in the SmartPortraits dataset.

The dataset includes metadata for each sequence, such as gender and the presence of glasses or facial hair, which often pose difficulties for neural rendering and reconstruction algorithms. By capturing subjects in various poses (standing, sitting, hands on hips), the dataset provides a robust benchmark for evaluating how different physical configurations affect 3D modeling accuracy.

Methodology and Technical Innovations

To bridge the gap between raw mobile data and high-accuracy ground truth, the researchers developed a custom handheld rig. This rig paired a Samsung S10e smartphone with an Azure Kinect DK depth camera. Achieving high-quality data required several technical breakthroughs:

  • Sub-millisecond Synchronization: Using a hybrid hardware/software approach and the "Twist-n-Sync" algorithm, the team ensured that smartphone RGB frames were perfectly aligned with the Kinect's high-accuracy depth maps.
  • Pseudo-Ground Truth Trajectories: Since traditional Motion Capture (MoCap) systems are limited to laboratory settings, the authors utilized high-end RGB-D SLAM to generate reference trajectories for outdoor and public scenes. This approach was validated against MoCap data, showing an error margin of only 1mm to 1cm.
  • Mutually Orthogonal Metric (MOM): To evaluate trajectory quality in the absence of ground truth, the authors employed the MOM metric. This no-reference metric assesses the "crispness" of the resulting point cloud, providing a quantitative measure of how well a SLAM algorithm has localized the camera.
Visualization of estimated camera trajectories relative to the reconstructed 3D point cloud of a subject.

Key Findings and Benchmarks

The researchers benchmarked several state-of-the-art (SOTA) algorithms, including ORB-SLAM and VINS-Mono, against the SmartPortraits data. Several key insights emerged:

  1. Visual vs. Visual-Inertial Performance: Surprisingly, purely visual methods often outperformed visual-inertial (VI) methods. This is attributed to the "instant-on" nature of handheld recording; most VI algorithms require a static initialization period to calibrate IMU biases, which is rarely present in casual smartphone use.
  2. Photometric Consistency vs. Geometric Precision: The study found that the most mathematically accurate camera trajectory does not always yield the most visually pleasing 3D reconstruction. For neural rendering (like NeRF), photometric consistency—how well colors align across different views—is often more critical than raw geometric precision.
  3. The Scale Factor: While methods like COLMAP are excellent at determining camera paths, they lack real-world scale. By incorporating external depth data, the SmartPortraits dataset provides the absolute scale necessary for accurate 3D reconstruction.

Conclusion and Future Work

SmartPortraits provides a massive volume of data—covering 6.6 km of trajectory over 11 hours of footage—to train and test the next generation of mobile 3D vision algorithms. While the dataset enables high-quality reconstruction using tools like ACMP and NeRF, it also highlights areas for improvement. Future research should focus on more robust IMU initialization for mobile devices and the joint optimization of camera poses and human micro-movements to further eliminate distortion in 3D portraits.

Summary updated

BibTeX
@inproceedings{kornilova2022smartportraits,
  author = {Anastasiia Kornilova and Marsel Faizullin and Konstantin Pakulev and Andrey Sadkov and Denis Kukushkin and Azat Akhmetyanov and Timur Akhtyamov and Hekmat Taherinejad and Gonzalo Ferrer},
  title = {{SmartPortraits: Depth Powered Handheld Smartphone Dataset of Human Portraits for State Estimation, Reconstruction and Synthesis}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.02063},
}
Figure