Abstract
In this work we tackle the problem of child engagement estimation while children freely interact with a robot in their room. We propose a deep-based multi-view solution that takes advantage of recent developments in human pose detection. We extract the child's pose from different RGB-D cameras placed elegantly in the room, fuse the results and feed them to a deep neural network trained for classifying engagement levels. The deep network contains a recurrent layer, in order to exploit the rich temporal information contained in the pose data. The resulting method outperforms a number of baseline classifiers, and provides a promising tool for better automatic understanding of a child's attitude, interest and attention while cooperating with a robot. The goal is to integrate this model in next generation social robots as an attention monitoring tool during various CRI tasks both for Typically Developed (TD) children and children affected by autism (ASD).
Introduction
In Child-Robot Interaction (CRI), maintaining a child's engagement is critical, particularly for therapeutic applications like social skills training for children with Autism Spectrum Disorder (ASD). However, measuring "engagement" automatically is difficult. Most existing systems require the child to remain stationary in front of a single sensor. When children move naturally, body parts become occluded, and single-camera views lose track of the child's posture and orientation.
The authors tackle the challenge of estimating engagement in a "free-play" environment where the child is not tethered or restricted to a specific seat, requiring a system that is robust to occlusions and capable of understanding temporal dynamics (how behavior changes over time).
Methodology
The proposed system utilizes a multi-stage pipeline designed for robustness in unconstrained environments.
Multi-View 3D Pose Estimation and Fusion
The setup employs four Kinect RGB-D cameras placed strategically around the room. The system extracts 2D keypoints using the OpenPose library and lifts them to 3D using depth data. To handle occlusions, all camera views are registered into a single coordinate system using the Iterative Closest Point (ICP) algorithm. This multi-view approach allows the system to interpolate or average data if one camera loses sight of a joint, ensuring continuous tracking even when a child turns their back to a sensor.
Feature Extraction and Temporal Classification
Beyond raw pose, the system tracks the Nao robot's head using a fine-tuned YOLOv3 network to calculate relative features, such as gaze direction and body facing angle relative to the robot. These features are fed into a deep neural network consisting of three Fully Connected (FC) layers followed by a Long Short-Term Memory (LSTM) layer. The LSTM is crucial for capturing the temporal evolution of engagement, distinguishing between steady focus and shifting attention.
Engagement is classified into three levels:
- Class 1 (Disengaged): Limited or no attention.
- Class 2 (Attentive): Observing but not actively cooperating.
- Class 3 (Actively Cooperating): Engaged in tasks like handovers.
Experimental Results
The study involved 25 children aged 6–10 years. Expert psychologists provided ground-truth labels for 1-second segments of interaction. The proposed 3FC+LSTM architecture outperformed traditional static classifiers like SVMs and Random Forests, achieving a Mean F-Score of 62.18% and an overall accuracy of 77.11%.
Key insights include:
- Temporal Context is King: The superiority of the LSTM model confirms that engagement is better understood through movement patterns over time rather than static snapshots.
- Multi-View Robustness: The four-camera setup maintained tracking even when children moved behind furniture, a common occurrence in natural child behavior.
- Relative vs. Absolute Features: Focusing on the child's pose relative to the robot makes the model invariant to specific room layouts or camera placements.
Conclusion and Future Work
While the current model shows promise with typically developed children, future work will focus on validating and fine-tuning the system for children with ASD, who may exhibit different social cues and movement patterns. Additionally, the researchers aim to integrate this offline estimation into a real-time, closed-loop system, allowing social robots to adapt their behavior dynamically to re-engage distracted children.
Summary updated