Toward an Attentive Robotic Architecture: Learning-Based Mutual Gaze Estimation in Human-Robot Interaction

Maria Lombardi, Elisa Maiettini, Davide De Tommaso, Agnieszka Wykowska, Lorenzo NataleFrontiers Robotics AI 2022 · DOI · OpenReview

TL;DR

This paper introduces a novel learning-based system for humanoid robots to automatically detect mutual gaze (eye contact) with humans during interactions. This is a crucial step towards creating more socially intelligent robots that can better understand and respond to human social cues, moving beyond passive observation to active engagement.

What Problem Does This Paper Address?

As robots increasingly move from controlled industrial settings into human environments for collaboration and assistance, they need to possess social intelligence to interact naturally. A key aspect of human-human interaction is "joint attention," where individuals share focus on the same object or event, often signaled by mutual gaze. While this is well-studied in human-human and human-screen interactions, its implementation in human-robot interaction (HRI) is less explored. Existing robotic systems often rely on pre-programmed behaviors or external, cumbersome hardware like eye-trackers. This paper addresses the challenge of enabling a humanoid robot to reliably and automatically detect mutual gaze using only its own onboard cameras, making HRI more natural and less reliant on external equipment.

Methods

The core of the proposed solution is a learning-based framework for eye contact detection.

  • Feature Extraction: The system utilizes OpenPose, a real-time multi-person pose estimation tool, to extract facial keypoints (eyes, nose, ears) from images captured by the robot's cameras. A feature vector is constructed from these keypoints, which is designed to be robust to variations in lighting and background.
  • Classification: A Support Vector Machine (SVM) with an RBF kernel is employed as the binary classifier. This model is trained to distinguish between "eye contact" and "no eye contact" based on the extracted facial keypoint features. The SVM was chosen over a random forest due to superior performance.
  • Data Augmentation: To improve robustness, the training data was augmented by simulating cases where OpenPose might fail to detect eye keypoints (setting them to zero) and by geometrically rotating the face keypoints to account for head movements.

Datasets

  • Newly Introduced Dataset: The paper introduces a novel dataset specifically collected for mutual gaze detection in frontal human-robot interaction. This dataset was gathered using the iCub robot's onboard cameras and an Intel RealSense depth camera. It consists of 484 frames (207 with eye contact, 277 without).
  • Training and Testing: The dataset was split into training and testing sets, with 19 participants for training and 5 for testing. The data was augmented to increase its size and robustness.

Key Insights and Ideas

  • Mutual Gaze as a Foundation for Social Attunement: The paper strongly emphasizes that mutual gaze is a fundamental social cue that signals readiness for interaction and attention, crucial for establishing natural social attunement between humans and robots.
  • Onboard Camera Reliance: A significant contribution is the demonstration that reliable mutual gaze detection can be achieved using only the robot's own eye-like cameras, eliminating the need for external eye-trackers or specialized hardware. This makes the interaction more natural and less intrusive.
  • Facial Keypoints as Robust Features: Using facial keypoints extracted by OpenPose as features proved to be effective and made the classifier independent of lighting conditions and background clutter.
  • Potential for Intent Prediction: The authors highlight that predicting human intention is a long-term goal, and gaze estimation is a critical first step towards this, as humans often look at an object before interacting with it.

Key Results

  • High Accuracy: The proposed learning-based approach achieved high performance metrics. On the collected test set, accuracy was around 91% ± 3%, with an F1 score of 89% ± 4%.
  • Temporal Sequence Performance: When evaluated on video sequences, the system demonstrated even higher accuracy, reaching 97% in some scenarios and 93% in others, with an F1 score of 98%.
  • Superior to State-of-the-Art: The proposed method outperformed the state-of-the-art approach by Chong et al. (2020) by approximately 15% in accuracy and 12% in F1 score on the test set, and 8% in accuracy and 6% in F1 score on video streams. This improvement is particularly notable given the challenging conditions (face masks, lower quality robot cameras) faced by the proposed method.
  • Model Interpretability: SHAP analysis indicated that internal eye keypoints and ear keypoints contribute most significantly to the prediction, with no single feature being entirely dominant.

Limitations and Future Work

  • Frontal Interaction Focus: The current system is primarily trained and tested for frontal interactions. Performance drops when the human's torso is rotated to extreme angles (90 degrees) while their head remains facing the robot.
  • Future Work: The authors suggest incorporating temporal information (e.g., temporal coherence between frames, optical flow) to further improve performance and generalization. They also aim to extend this to a more comprehensive attention system capable of detecting a wider range of social cues, moving towards truly socially capable humanoid robots. The development of an "attentive architecture" is the overarching goal.

Summary updated

Abstract

p>Social robotics is an emerging field that is expected to grow rapidly in the near future. In fact, it is increasingly more frequent to have robots that operate in close proximity with humans or even collaborate with them in joint tasks. In this context, the investigation of how to endow a humanoid robot with social behavioral skills typical of human–human interactions is still an open problem. Among the countless social cues needed to establish a natural social attunement, this article reports our research toward the implementation of a mechanism for estimating the gaze direction, focusing in particular on mutual gaze as a fundamental social cue in face-to-face interactions. We propose a learning-based framework to automatically detect eye contact events in online interactions with human partners. The proposed solution achieved high performance both <italic>in silico</italic> and in experimental scenarios. Our work is expected to be the first step toward an attentive architecture able to endorse scenarios in which the robots are perceived as social partners.</p>
BibTeX
@article{lombardi2022toward,
  author = {Maria Lombardi and Elisa Maiettini and Davide De Tommaso and Agnieszka Wykowska and Lorenzo Natale},
  title = {{Toward an Attentive Robotic Architecture: Learning-Based Mutual Gaze Estimation in Human-Robot Interaction}},
  journal = {Frontiers Robotics AI},
  year = {2022},
  doi = {10.3389/frobt.2022.770165},
}
Figure