TL;DR
This paper systematically compares various depth map representations (depth images, normal images, point clouds, voxels) and deep learning architectures for face recognition. It finds that normal images and point clouds offer superior generalization capabilities, especially across different sensors and acquisition settings, and introduces a new challenging dataset, MultiSFace, to further investigate these cross-domain challenges.
What Problem Does This Paper Address?
The increasing availability of active depth sensors has opened new avenues for computer vision tasks like face recognition. However, a significant hurdle is the poor generalization of deep learning models when faced with depth data from different sensor technologies (e.g., Structured Light vs. Time-of-Flight) or varying acquisition conditions (distance, lighting). Existing datasets are often limited in scope, acquired with a single sensor type, and lack diversity in acquisition settings, making it difficult to assess how well models trained on one dataset will perform on another. This paper aims to bridge this gap by providing a comprehensive comparison of depth map representations and deep learning approaches to identify configurations that maximize both accuracy and generalization.
Methods
The authors conducted a thorough experimental analysis comparing several key components:
- Depth Data Representations: They evaluated four primary ways to represent depth information:
- Depth Images: Standard 2D grayscale images where pixel intensity encodes depth.
- Normal Images: Images representing surface normals (direction perpendicular to the surface) derived from depth data.
- Point Clouds: Unordered sets of 3D points representing the scene geometry.
- Voxels: 3D grids where each cell (voxel) indicates occupancy.
- Deep Learning Architectures: They employed established deep learning models suitable for each representation: 2D CNNs (VGG-16, ResNet, Inception) for images, PointNet and PointNet++ for point clouds, and VoxNet, R3D, and R(2+1)D for voxels.
- Pre-processing and Normalization Techniques: Various methods like filtering, histogram equalization, hole filling, and different normalization strategies were applied to depth images and point clouds to assess their impact.
- Sensor Technologies: The experiments considered data from both Structured Light (SL) and Time-of-Flight (ToF) depth sensors.
- Acquisition Settings: The influence of subject-to-sensor distance and sensor resolution was investigated.
The study involved both intra-dataset experiments (training and testing on the same dataset) to evaluate baseline performance and cross-dataset experiments to assess generalization across different sensors and acquisition conditions.
Datasets
The paper utilized four publicly available depth-based face recognition datasets:
- Biwi [22]: Acquired with a first-generation Microsoft Kinect (SL).
- CurtinFaces [28]: Also acquired with a first-generation Microsoft Kinect (SL).
- Lock3DFace [21]: Acquired with a second-generation Microsoft Kinect (ToF).
- Pandora [14]: Acquired with a second-generation Microsoft Kinect (ToF).
Additionally, the authors introduced a novel dataset:
- MultiSFace: This dataset was specifically created to evaluate cross-device and cross-distance face recognition. It includes data from two synchronized depth sensors (high-resolution ToF and low-resolution ToF) at two different distances (1m and 2.5m) for 31 subjects. This dataset is designed to be a challenging benchmark for generalization.
Key Insights and Ideas
- Depth Images are Not Enough: Treating depth maps simply as 2D grayscale images for CNNs limits their generalization. While effective in intra-dataset scenarios, they struggle when faced with data from different sensors or acquisition settings.
- Surface Normals Offer Invariance: Normal images, derived from depth data, capture intrinsic 3D shape information that is more invariant to sensor characteristics and acquisition conditions compared to raw depth images. This leads to better generalization.
- Point Clouds Shine in Generalization: Point clouds, despite being less commonly used for face recognition, demonstrated the best generalization capabilities, particularly in cross-dataset and cross-device scenarios. This suggests they are more robust to variations in sensor technology and data quality.
- The Challenge of "In the Wild" Depth Face Recognition: The MultiSFace dataset highlights the significant difficulty of achieving high accuracy in real-world scenarios with diverse sensors and acquisition settings. Performance drops considerably compared to controlled intra-dataset experiments.
- Pre-processing Nuances: While common pre-processing techniques like filtering and equalization are often applied, their benefits are not always clear and can sometimes reduce the 3D content, potentially hindering generalization. Data normalization, however, generally maintained or improved results.
Key Results
- Intra-Dataset Performance: For intra-dataset tests, normal images combined with CNNs (specifically ResNet) achieved the highest accuracy across most datasets.
- Cross-Dataset Performance: In cross-dataset experiments, point clouds (especially with PointNet++) showed the most promising generalization, outperforming depth images and voxels. Normal images also performed well with CNNs.
- MultiSFace Performance: Experiments on the MultiSFace dataset revealed a significant performance drop for all tested methods, especially for image-based approaches (4-6% accuracy). Voxel-based methods showed some resilience to resolution differences at closer distances.
- Computational Complexity: 2D CNNs like VGG-16 have high parameter counts and memory usage but fast inference. PointNet++ exhibited a very high inference time, likely due to unoptimized clustering operations.
Limitations and Future Work
- Low Accuracy on MultiSFace: The authors acknowledge that the accuracy achieved on the MultiSFace dataset is quite low, indicating that robust face recognition across diverse depth sensors and acquisition settings is still an open challenge.
- Point Cloud Generalization Discrepancy: The paper notes a potential difficulty in assessing point cloud methods due to a lack of extensive cross-dataset studies in the literature, suggesting their reported results can serve as a baseline.
- Future Investigations: The MultiSFace dataset is presented as a valuable resource for future research focused on developing face recognition methods that are invariant to acquisition sensors and settings, and that can fully exploit the 3D content of depth maps. The authors suggest that future work should focus on architectures and representations that are inherently more robust to these variations.
Summary updated