TL;DR
This paper introduces NITEC, a large-scale, hand-annotated dataset for eye contact detection in ego-vision scenarios, addressing the critical need for robust datasets to train machines for this fundamental human interaction. The authors demonstrate that NITEC enables better generalization for eye contact detection models compared to existing datasets, paving the way for more intuitive human-robot interactions.
What Problem Does This Paper Address?
Eye contact is a cornerstone of human social interaction, playing a vital role in communication, regulating social dynamics, and establishing joint attention. While humans are remarkably adept at perceiving eye contact, even in challenging conditions, machines still struggle to reliably detect it. This is particularly problematic in human-robot interaction (HRI), where eye contact can significantly enhance a robot's likability and perceived human-likeness. The authors argue that a primary reason for this gap is the lack of rich, high-quality datasets that capture the nuances of eye contact in unconstrained, ego-centric (first-person) perspectives. Existing datasets are often too small, lack diversity in demographics and scenarios, or are not specifically curated for this task, hindering the development of robust eye contact detection models.
Methods
The core contribution of this paper is the creation and release of the NITEC dataset. NITEC is built by combining and re-annotating data from four existing datasets: WIDER FACE, Gaze360, CelebA, and Helen. This approach leverages the strengths of each source: WIDER FACE for its large scale and diverse scene contexts, Gaze360 for its extensive gaze estimation data, CelebA for its feature-rich faces and varied gaze directions, and Helen for its diverse appearance variations.
The annotation process involved using a dedicated tool that highlighted face crops, allowing annotators to consider context outside the immediate facial region. Annotators subjectively labeled whether a face appeared to be making eye contact with the camera. For the training set, two annotators were used, while the test set was labeled by three annotators, with the majority vote determining the final label. The dataset was split into approximately 80% for training and 20% for testing.
For evaluation, the authors used standard CNN architectures like ResNet18 and ResNet50, as well as the SWIN-Transformer. They employed average precision (AP) and F1-Score as primary metrics for quantitative analysis. Qualitative analysis involved visualizing activation maps and prediction distributions to understand model behavior.
Datasets
- NITEC (New Dataset Introduced): A large-scale, hand-annotated dataset for ego-vision eye contact detection, comprising 35,919 samples. It's a combination and re-annotation of:
- WIDER FACE: Primarily for face detection, contributing images with varied scene contexts and smaller face resolutions.
- Gaze360: For gaze estimation, providing 3D gaze vectors that were converted to 2D.
- CelebA: A large-scale celebrity face dataset, offering images with challenging gaze angles.
- Helen: A dataset for face feature analysis, providing diverse appearance variations.
- Existing Datasets for Comparison:
- OFDIW: A publicly available dataset for eye contact detection.
- DEEPEC: Another publicly available dataset for eye contact detection.
- MPIIFaceGaze: Used for qualitative analysis of prediction distributions.
Key Insights and Ideas
- Dataset Diversity is Crucial: Combining datasets with complementary characteristics (scene context, face resolution, gaze variation) leads to a more robust and versatile eye contact detection dataset (NITEC).
- Hand-Annotation Matters: Manual annotation, even with subjective decisions, provides a higher quality signal for eye contact compared to relying solely on automated gaze estimation or head pose.
- Head Pose and Gaze Estimation are Insufficient: Models trained solely on head pose or gaze direction struggle to accurately detect eye contact, especially in unconstrained scenarios. Their prediction errors and variance are too high.
- Simpler Architectures Can Excel: The NITEC dataset is rich enough to enable good generalization performance even with simpler baseline models like ResNet18, suggesting the dataset itself is a key enabler.
- Dataset Noise Impacts Performance: Datasets with significant label noise (like OFDIW, as suggested by the authors) can lead to poor model performance, even when using the same model architecture.
Key Results
- NITEC's Superiority: Models trained on NITEC consistently outperformed models trained on other datasets when evaluated on various test sets, demonstrating superior cross-dataset generalization. For instance, ResNet18 trained on NITEC achieved an AP of 88.9% on the NITEC test set, significantly higher than other models.
- Quantitative Performance:
- ResNet18 trained on NITEC achieved an AP of 88.9% and an F1-Score of 83.6% on its own test set.
- Head pose models (6DRepNet) and gaze direction models (L2SC-Net, Gaze360) showed significantly lower F1-Scores compared to direct eye contact classification models, even when thresholds were adjusted.
- The model proposed by Chong et al. [6], trained on a very large dataset, showed robust performance but still fell behind NITEC-trained models on the NITEC test set.
- Qualitative Findings: Visualizations showed that ResNet18, trained on NITEC, was better at integrating broader facial context and social cues compared to ResNet50, which sometimes over-relied on ancillary attributes. NITEC also demonstrated more conservative and less noisy predictions compared to other methods.
Limitations and Future Work
- Limitations:
- The authors acknowledge that while NITEC aims to reduce false positives by including WIDER FACE, this can sometimes lead to false negatives in challenging cases (e.g., blurred faces).
- Larger SWIN-Transformer architectures (small and base) might require more training data than currently available in NITEC to reach their full potential.
- The paper suggests that the OFDIW dataset might suffer from significant label noise, impacting model performance.
- Future Work:
- The authors plan to further investigate the observed behavior and link it with a dedicated user study to understand the subjective perception of eye contact.
- They aim to foster reproducibility and further exploration in the field of ego-vision interaction by making the NITEC dataset publicly available.
Summary updated