Transfer Learning of Engagement Recognition within Robot-Assisted Therapy for Children with Autism

Nazerke Rakhymbayeva, Anara SandygulovaAAAI 2021 · DOI

Introduction

Robot-Assisted Therapy (RAT) has emerged as a promising tool for supporting children with Autism Spectrum Disorder (ASD). Robots act as social mediators, helping children develop communication and social skills in a controlled, predictable environment. However, evaluating the success of these interventions currently relies on "video coding"—a manual process where experts spend hours annotating child behavior. This labor-intensive bottleneck prevents real-time adaptation and large-scale clinical deployment. This research aims to automate engagement recognition using multi-modal data and transfer learning, creating a system that is objective, scalable, and capable of generalizing across diverse individuals.

The Challenge: Heterogeneity and Data Scarcity

Automating engagement recognition for children with ASD presents two primary hurdles. First is the heterogeneity of autism: children on the spectrum express engagement through highly individualized and often non-typical behavioral cues. A model trained on one child may fail to recognize engagement in another. Second is data scarcity: high-quality, annotated datasets of children with ASD are rare due to ethical and privacy constraints.

To address these, this project investigates transfer learning—the process of taking knowledge gained from a "source domain" (larger datasets of typically developing populations) and applying it to a "target domain" (the specific ASD population).

Methodology

The proposed framework utilizes a multi-stage pipeline designed for robustness and clinical utility:

1. Multi-Modal Feature Extraction

The system uses the OpenPose library to extract 2D keypoints from video frames. By capturing 25 body/foot joints, 42 hand points, and 70 facial keypoints, the model builds a comprehensive digital representation of the child's physical presence and expressions. This multi-modal approach ensures that the system doesn't rely solely on facial expressions, which can be subtle or atypical in children with ASD.

2. Classification Architecture

The core of the recognition system is a neural network utilizing ReLU and Softmax activation functions within its hidden layers. To predict engagement levels on a granular 0 to 5 scale, the final output layer employs a sigmoid function with cross-entropy loss.

3. Validation Strategy

To ensure the model's practical applicability, the researchers use "Child Independent" splits. This means the model is tested on data from children it has never encountered during training, simulating a real-world clinical scenario where a robot must interact with a new patient.

Data Landscape

The research leverages a combination of a substantial new dataset and established public benchmarks:

  • Primary ASD Dataset: This includes behavioral data from 36 children with ASD across 194 therapy sessions, totaling over 48 hours of video. Two independent raters achieved an 82.6% inter-rater reliability score during manual annotation.
  • Public Benchmarks: To facilitate transfer learning, the study incorporates the DREAM dataset (robot-enhanced ASD therapy), PInSoRo (social dynamics), and MHHRI (multimodal human-human-robot interaction).

Key Findings

The experimental results highlight both the potential and the complexity of automated engagement recognition:

  • Binary vs. Multi-class Performance: The model achieved a respectable 79.34% accuracy in binary classification (engaged vs. not engaged). However, performance dropped to 47.64% when attempting to classify engagement on the more granular 0-5 scale. This "difficulty spike" underscores the subtle nuances involved in clinical engagement levels.
  • Modality Synergy: The research confirms that combining facial features with body pose is essential. Relying on a single modality often fails to capture the diverse expressive styles inherent in the autistic population.
  • Feature Transferability: A critical insight of the study is that not all social cues are universal. Identifying which features are truly transferable from typical populations to the ASD population is the key to creating generalizable Human-Robot Interaction (HRI) systems.

Conclusion and Future Work

While binary engagement recognition shows promise, achieving high-granularity accuracy across different therapeutic activities (such as storytelling versus dancing) remains a significant challenge. Future efforts will focus on Optimal Feature Selection to identify the specific behavioral cues that remain predictive across different scenarios and individuals. By refining these transfer learning techniques, the goal is to provide therapists with a real-time, objective tool that enhances the effectiveness of robot-assisted interventions for children with autism.

Summary updated

BibTeX
@inproceedings{rakhymbayeva2021transfer,
  author = {Nazerke Rakhymbayeva and Anara Sandygulova},
  title = {{Transfer Learning of Engagement Recognition within Robot-Assisted Therapy for Children with Autism}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17861},
}
Figure