Early detection of autism using digital behavioral phenotyping

Sam Perochon, J. Matías Di Martino, Kimberly L. H. Carpenter, Scott N. Compton, Naomi Davis, Brian Eichner, Steven Espinosa, Lauren Franz, Pradeep Raj Krishnappa Babu, Guillermo Sapiro, Géraldine DawsonNature Medicine 2023 · DOI

TL;DR

Researchers have developed "SenseToKnow," a tablet-based app that uses computer vision and machine learning to detect autism in toddlers during routine primary care visits. By capturing 23 distinct "digital phenotypes"—from gaze patterns to head movements—the tool achieves high diagnostic accuracy (AUC = 0.90) and, crucially, maintains consistent performance across sex, race, and ethnicity, addressing long-standing biases in traditional screening.

What Problem Does This Paper Address?

Early intervention is the "gold standard" for supporting children with autism, but the path to a diagnosis often starts with a bottleneck: the screening questionnaire. The current industry standard, the M-CHAT-R/F, relies on parent reports which can be subjective. In real-world primary care settings, its sensitivity often plummets to around 39%, and it frequently misses signs in girls and children of color.

While specialized eye-tracking hardware has shown promise in research labs, it is expensive and difficult to scale. There is a desperate need for an objective, scalable, and equitable tool that can be used in a busy pediatrician's office to identify children who need a full diagnostic evaluation.

Methods

The authors developed the SenseToKnow app, which runs on a standard iPad. While a child watches brief, strategically designed movies (e.g., social scenes vs. nonsocial objects) and plays a bubble-popping game, the front-facing camera records their behavior.

The technical pipeline is sophisticated:

  1. Computer Vision Analysis (CVA): The system automatically detects the child's face and extracts 49 facial landmarks, head pose (pitch, yaw, roll), and gaze coordinates at 30 frames per second.
  2. Digital Phenotyping: It quantifies 23 behavioral features, including social attention, facial expression complexity, blink rate, response to name (triggered by the examiner), and motor skills (via touch-screen data).
  3. Machine Learning: An ensemble of 1,000 XGBoost (Extreme Gradient Boosting) trees integrates these phenotypes to classify the child.
  4. Interpretability: The team used SHAP (SHapley Additive exPlanations) values to provide "explainable AI." This allows clinicians to see exactly which behaviors (e.g., a delay in responding to their name or a preference for nonsocial stimuli) contributed to a specific child's score.

Datasets

The study was a multiclinic, prospective trial involving 475 toddlers (aged 17–36 months) recruited during well-child visits at Duke University Health System.

  • Autism group: 49 children.
  • Developmental Delay (without autism) group: 98 children.
  • Neurotypical group: 328 children.
  • Demographics: The sample was diverse, including 269 boys and 206 girls, with significant representation of Black (8.5%) and Hispanic/Latino (6.7%) children.

Key Insights and Ideas

  • Multimodal Strength: The study found that while individual features (like gaze) are helpful, the combination of 23 phenotypes is what drives high accuracy. Autism is heterogeneous; one child might show atypical gaze, while another might show reduced facial mimicry. The ML model captures this complexity.
  • Closing the Equity Gap: Perhaps the most significant finding is that the app performed consistently across sex, race, and ethnicity. Traditional questionnaires often miss girls because they may present with different social behaviors; the digital phenotypes used here were robust enough to bypass these biases.
  • The Power of "And": The app isn't meant to replace parent questionnaires but to augment them. When the app's results were combined with the M-CHAT-R/F, the diagnostic accuracy (AUC) climbed to a remarkable 0.97.
  • Clinical Utility Metrics: The researchers introduced "Quality Scores" (to ensure the app was administered correctly) and "Confidence Scores" (to tell the doctor how certain the algorithm is). This makes the tool practical for a non-expert to use.

Key Results

  • Overall Accuracy: The algorithm achieved an AUC of 0.90, with 87.8% sensitivity and 80.8% specificity.
  • Predictive Value: The Negative Predictive Value (NPV) was 97.8%, meaning the app is excellent at "ruling out" autism, which is critical for a screening tool. The Positive Predictive Value (PPV) was 40.6%, significantly higher than the ~15% typically seen with traditional questionnaires in primary care.
  • Top Predictors: SHAP analysis revealed that "Facing forward during social movies" and "Gaze percent social" were the most influential features in the model.

Limitations and Future Work

The authors acknowledge a few key hurdles:

  • Validation Bias: While the autism and developmental delay groups received full diagnostic workups, the "neurotypical" group was largely identified via electronic health records and lack of parent concern, which could lead to some missed cases.
  • Sample Prevalence: The study had a higher-than-average prevalence of autism (roughly 10% vs. the 2% in the general population), which can inflate certain performance metrics.
  • Feature Scope: The app currently focuses on social communication and motor skills but does not yet capture "restricted and repetitive behaviors," a core diagnostic criterion for autism.

Future work will involve testing the app in even larger, more diverse populations and refining the algorithms to detect more subtle behavioral nuances.

Summary updated

Abstract

Early detection of autism, a neurodevelopmental condition associated with challenges in social communication, ensures timely access to intervention. Autism screening questionnaires have been shown to have lower accuracy when used in real-world settings, such as primary care, as compared to research studies, particularly for children of color and girls. Here we report findings from a multiclinic, prospective study assessing the accuracy of an autism screening digital application (app) administered during a pediatric well-child visit to 475 (17-36 months old) children (269 boys and 206 girls), of which 49 were diagnosed with autism and 98 were diagnosed with developmental delay without autism. The app displayed stimuli that elicited behavioral signs of autism, quantified using computer vision and machine learning. An algorithm combining multiple digital phenotypes showed high diagnostic accuracy with the area under the receiver operating characteristic curve = 0.90, sensitivity = 87.8%, specificity = 80.8%, negative predictive value = 97.8% and positive predictive value = 40.6%. The algorithm had similar sensitivity performance across subgroups as defined by sex, race and ethnicity. These results demonstrate the potential for digital phenotyping to provide an objective, scalable approach to autism screening in real-world settings. Moreover, combining results from digital phenotyping and caregiver questionnaires may increase autism screening accuracy and help reduce disparities in access to diagnosis and intervention.
BibTeX
@inproceedings{perochon2023early,
  author = {Sam Perochon and J. Matías Di Martino and Kimberly L. H. Carpenter and Scott N. Compton and Naomi Davis and Brian Eichner and Steven Espinosa and Lauren Franz and Pradeep Raj Krishnappa Babu and Guillermo Sapiro and Géraldine Dawson},
  title = {{Early detection of autism using digital behavioral phenotyping}},
  booktitle = {Nature Medicine},
  year = {2023},
  doi = {10.1038/s41591-023-02574-3},
}
Figure