Attention-Based Autism Spectrum Disorder Screening With Privileged Modality

Shi Chen, Qi Zhao

Introduction

Autism Spectrum Disorder (ASD) affects approximately 1 in 59 children in the United States. Despite its prevalence, clinical screening remains a significant bottleneck due to the subjective nature of current diagnostic methods, which are time-consuming and require highly specialized personnel. While computer vision has been explored for gaze tracking in ASD screening, existing approaches often overlook temporal fixation dynamics or necessitate expensive, specialized equipment.

This research addresses three critical gaps: the scarcity of objective automated tools, the reliance on static heatmap analysis rather than temporal sequences, and the requirement for multi-modal data at inference time. The authors propose a framework that leverages "privileged information"—using one data modality during training to enhance the performance of a model processing a different modality during screening.

Methodology

The proposed framework integrates two deep neural network (DNN) models and a novel multi-modal distillation strategy.

Task-Specific Models

  1. Photo-Taking Task (Encoder-PT): Participants freely explore an environment and capture photographs. A ResNet-50 combined with a Long Short-Term Memory (LSTM) network analyzes these images. This task serves as an ecological proxy for social interaction, capturing preferences for objects over social subjects or atypical compositional styles.
  2. Image-Viewing Task (Encoder-IV): Participants view images while their eye movements are recorded. Unlike traditional static heatmap methods, this model uses an LSTM to process the temporal sequence of fixations, capturing the dynamic nature of visual attention.

Multi-Modal Distillation

The core innovation is a three-phase training process designed to transfer knowledge between modalities. Initially, models are trained independently. In the second phase, both models project features into a shared embedding space to learn commonalities between photo-taking and gaze patterns. Finally, in Phase III, this shared knowledge is distilled back into the individual models.

The multi-modal distillation process (Phase III) transfers knowledge from a shared embedding space back to individual task encoders.

This approach allows the photo-taking model to benefit from gaze-pattern insights even when eye-tracking data is unavailable during the actual screening process.

Key Results

The framework demonstrates superior performance across both tasks, often surpassing human expert benchmarks.

Gaze-Based Screening

On the image-viewing task, the model achieved 100% sensitivity and 95% specificity, setting a new state-of-the-art for gaze-based ASD identification. The inclusion of temporal dynamics via LSTM proved crucial, as the order of fixations was found to be highly discriminative.

Comparison of gaze heatmaps between ASD and control subjects across different image stimuli.

Photo-Taking Performance

The AI achieved 84% accuracy in identifying ASD from participant-taken photos. Notably, human experts (trained clinicians) only achieved 65% accuracy on the same dataset. The "Privileged Modality" distillation significantly boosted performance; the photo-taking model's accuracy improved from 76% (when trained independently) to 84% after distillation from the gaze-data shared space.

Discussion and Conclusion

The study highlights that temporal gaze patterns and ecological tasks like photo-taking provide rich, objective markers for ASD. The ability to transfer knowledge between different groups of participants (those who provided gaze data vs. those who took photos) makes this framework highly practical for clinical settings where multi-modal data collection for every patient is unfeasible.

Future work will focus on expanding the sample size beyond the current pilot cohort and exploring the use of standard webcams for gaze estimation to further reduce hardware barriers. Additionally, increasing the interpretability of the deep learning models will be essential for clinical adoption, helping practitioners understand the specific visual features driving the AI's classifications.

Summary updated

Abstract

This paper presents a novel framework for automatic and quantitative screening of autism spectrum disorder (ASD). It is motivated to address two issues in the current clinical settings: 1) short of clinical resources with the prevalence of ASD (1.7% in the United States), and 2) subjectivity of ASD screening. This work differentiates itself with three unique features: first, it proposes an ASD screening with privileged modality framework that integrates information from two behavioral modalities during training and improves the performance on each single modality at testing. The proposed framework does not require overlap in subjects between the modalities. Second, it develops the first computational model to classify people with ASD using a photo-taking task where subjects freely explore their environment in a more ecological setting. Photo-taking reveals attentional preference of subjects, differentiating people with ASD from healthy people, and is also easy to implement in real-world clinical settings without requiring advanced diagnostic instruments. Third, this study for the first time takes advantage of the temporal information in eye movements while viewing images, encoding more detailed behavioral differences between ASD people and healthy controls. Experiments show that our ASD screening models can achieve superior performance, outperforming the previous state-of-the-art methods by a considerable margin. Moreover, our framework using diverse modalities demonstrates performance improvement on both the photo-taking and image-viewing tasks, providing a general paradigm that takes in multiple sources of behavioral data for a more accurate ASD screening. The framework is also applicable to various scenarios where one-to-one pairwise relationship is difficult to obtain across different modalities.
BibTeX
@inproceedings{chen2019attention,
  author = {Shi Chen and Qi Zhao},
  title = {{Attention-Based Autism Spectrum Disorder Screening With Privileged Modality}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00127},
}
Figure