TL;DR
This paper introduces an end-to-end system that uses facial attributes like expressions, action units, arousal, and valence to classify Autism Spectrum Disorder (ASD). By training a multi-task convolutional neural network on large-scale facial datasets and then applying temporal analysis to video inputs, the system achieves a significant improvement in ASD classification accuracy, reaching a 76% F1 score.
What Problem Does This Paper Address?
Diagnosing Autism Spectrum Disorder (ASD) can be challenging due to the wide spectrum of symptoms and the influence of non-specific factors like cognitive ability and age. While facial attributes have shown promise as markers for ASD in clinical studies, applying advanced machine learning techniques like Convolutional Neural Networks (CNNs) directly to ASD classification is hindered by the lack of large, labeled datasets specifically for ASD with these attributes. This paper aims to bridge this gap by developing a system that leverages existing large-scale facial datasets to extract relevant facial attributes and then uses these attributes for ASD classification.
Methods
The core of the system is an end-to-end machine learning pipeline. First, a CNN is trained to simultaneously predict four facial attributes: facial expressions, Action Units (AUs), arousal, and valence. This multi-task learning approach is employed to leverage shared representations and improve efficiency. The CNN is trained on a combination of two large-scale facial datasets.
Once the CNN is trained, it's used to extract a feature vector for each frame of a participant's video. These frame-level features are then processed using temporal analysis. This involves calculating the mean and standard deviation of the feature vectors across all frames, along with specific metrics for action units (mean activation time) and arousal/valence (percentage of positive frames). This temporal analysis condenses the video into a single, lower-dimensional feature vector per participant. Finally, this feature vector is fed into a binary classifier to predict whether the participant has ASD.
Datasets
- Training/Attribute Recognition: The authors combined two large-scale facial datasets:
- AffectNet [6]: Used for facial expressions, arousal, and valence.
-
EmotioNet [11]: Used for Action Units (AUs). The combined dataset contains approximately 1.2 million samples.
-
ASD Classification: A custom video dataset was collected, consisting of 105 children (62 with ASD, 43 without). Participants watched a video stimulus on an iPad, and their facial responses were recorded. 88 participants (49 ASD, 39 non-ASD) consented to use their data for research. Diagnostic labels were provided by clinicians.
Key Insights and Ideas
- Multi-task learning for facial attributes is superior: The study found that training a single CNN to predict multiple facial attributes simultaneously (multi-task learning) generally outperforms training separate CNNs for each attribute, especially for arousal and valence.
- Leveraging "in-the-wild" data is crucial: The success of the system hinges on training the attribute recognition CNN on large, diverse datasets of natural images ("in the wild"), which allows the network to learn robust representations that generalize well.
- Temporal analysis adds significant value: Simply using frame-level features is insufficient. Temporal analysis, which captures the dynamics of facial attributes over time, is essential for effective ASD classification.
- Specific facial attributes matter: The research demonstrates that incorporating arousal, valence, and facial expressions significantly improves ASD classification performance, highlighting their importance as biomarkers.
- Efficient CNN architectures are key: For facial attribute recognition, more complex CNN architectures (like Bottleneck) didn't necessarily yield proportional performance gains compared to more efficient ones (like EESP), suggesting that simpler models can be sufficient for this task.
Key Results
- Facial Attribute Recognition: The multi-task learning approach achieved better performance than single-task learning for most facial attributes, with an 8% improvement for arousal. The EESP CNN unit showed comparable performance to Bottleneck and MobileNet units but was significantly more efficient in terms of parameters and FLOPs.
- ASD Classification: The proposed system, using temporal analysis of facial attributes, achieved a significant improvement in ASD classification. Adding features related to arousal, valence, and facial expressions improved performance by approximately 7%. The final ASD classification achieved an F1 score of 76%.
- Statistical Significance: Student's t-tests confirmed that arousal, valence, and expression attributes were statistically significant predictors of ASD.
Limitations and Future Work
- Dataset Size for ASD Classification: The dataset for ASD classification is relatively small (88 participants), which limits the complexity of models that can be trained directly for this task. The authors mention that temporal features learned using RNNs or temporal CNNs might exhibit poor generalizability on their dataset due to this limitation.
- Temporal Feature Learning: While temporal analysis was performed, the paper notes that methods like RNNs and temporal CNNs could also be explored for learning temporal features, though they require more data.
- Generalizability of Attribute Recognition Models: While the attribute recognition models were trained on large datasets, their performance on the specific ASD video dataset was evaluated using leave-one-out cross-validation due to the limited size of the ASD dataset.
The authors suggest that their multi-task learning approach for facial attribute recognition is effective and that these attributes are significant for ASD classification. Future work could focus on collecting larger, more diverse ASD datasets to further refine these models and explore more sophisticated temporal modeling techniques.
Summary updated