Activity-Based Early Autism Diagnosis Using a Multi-Dataset Supervised Contrastive Learning Approach

Asha Rani, Yashaswi VermaWACV 2024

TL;DR

This paper introduces a novel multi-dataset supervised contrastive learning (MSupCL) approach to improve early autism diagnosis from video activity clips. By leveraging data from multiple, diverse datasets, MSupCL learns more robust and discriminative features, significantly outperforming existing methods on a challenging dataset.

What Problem Does This Paper Address?

Diagnosing Autism Spectrum Disorder (ASD) early is crucial for effective intervention, but traditional methods are time-consuming and may lead to delays. While automated diagnosis using video data shows promise, it faces significant challenges: limited training data, subtle visual differences between ASD and control subjects, and difficulty in identifying universally beneficial features. Existing machine learning approaches often struggle with these issues, especially when dealing with data from different sources with varying distributions. This paper aims to bridge this gap by developing a method that can learn effectively from multiple, independently collected video datasets, even when they have different distributions.

Methods

The core of this paper is the Multi-Dataset Supervised Contrastive Learning (MSupCL) approach. It builds upon the principles of contrastive learning, which aims to pull similar data points closer together and push dissimilar ones apart in a learned feature space.

Here's a breakdown of their method:

  • Multi-Dataset Integration: Unlike previous methods that focus on a single dataset or require paired multi-modal data, MSupCL is designed to work with multiple, unpaired datasets of the same modality (videos in this case) for the same task.
  • Supervised Contrastive Learning: MSupCL leverages category labels (ASD vs. Control) to create "positive" and "negative" pairs. For a given video clip (the "anchor"), a positive pair is formed with another clip from a different dataset belonging to the same category. A negative pair is formed with a clip from the other dataset belonging to a different category. This cross-dataset pairing is key to learning generalized features.
  • Feature Extraction Network: A deep encoder network (specifically, a ResNet-based R(2+1)D-18) is used to extract initial feature representations from the video clips. These are then passed through a multi-layer perceptron (MLP) head to generate embeddings.
  • MSupCL Loss Function: A novel loss function is proposed that generalizes supervised contrastive loss to the multi-dataset setting. It encourages embeddings of positive pairs to be close and negative pairs to be far apart.
  • Two-Step Training: First, the MSupCL approach is used to train the feature encoder network. Second, a dataset-specific classifier (a fully-connected layer with softmax) is trained on top of the frozen encoder for prediction.

Datasets

The authors used two publicly available activity video datasets for their experiments:

  • Hand Gesture Dataset [32]: This dataset contains videos of subjects performing four activities. It's described as challenging due to high intra-class variability and low inter-class variability, making it difficult to distinguish between ASD and Control subjects.
  • Autism Dataset [19]: This dataset features eight activities and is noted for having well-separated distributions between ASD and Control classes, making it an "easier" dataset.

Key Insights and Ideas

  • Leveraging "Easy" Data to Improve "Hard" Data: A significant insight is that knowledge from an "easier" dataset (Autism dataset) can be effectively transferred to improve performance on a "difficult" dataset (Hand Gesture dataset) through multi-dataset contrastive learning. This is akin to a form of "data-level knowledge distillation."
  • Unpaired Multi-Dataset Learning is Viable: The paper demonstrates that it's possible to learn effectively from multiple datasets that are not explicitly paired, as long as they share the same task and categories. This broadens the applicability of contrastive learning.
  • Visual Complexity Matters: The performance drop of MSSCL on their task highlights that self-supervised learning might not directly benefit from unpaired multi-source data when the visual complexity and low inter-class variability are high, as seen in the Hand Gesture dataset.
  • Categorical Supervision is Key for Cross-Dataset Pairing: While the approach is supervised, the supervision comes from category labels to create pairs across datasets, not from explicit sample-to-sample correspondence.

Key Results

  • Hand Gesture Dataset: MSupCL achieved the highest accuracy, outperforming the next best method (SSCL) by an absolute 13%. This dataset is considered challenging, and the results strongly validate the MSupCL approach.
  • Autism Dataset: All compared methods achieved near-perfect accuracy, indicating that the dataset's well-separated classes make it less challenging. MSupCL was comparable to the best approaches here.
  • Baseline Performance: A simple binary classifier performed poorly on the Hand Gesture dataset, highlighting the need for more sophisticated feature learning.
  • Qualitative Analysis: Visualizations (t-SNE) showed better separation of ASD and Control samples for MSupCL on the Hand Gesture dataset. Confusion matrices indicated MSupCL correctly classified more Control samples than other methods. Qualitative examples also showed MSupCL often had higher confidence in correct predictions and lower confidence in incorrect ones compared to SSCL.

Limitations and Future Work

  • False Positive/Negative Rates: Despite improvements, the paper acknowledges that false-positive and false-negative rates remain substantially high, indicating that further research is needed for deployable systems.
  • Computational Cost: The experiments consumed energy, a point the authors raise regarding the environmental impact of computational research.
  • Future Directions: The authors suggest that their MSupCL approach can be adapted to other modalities by simply plugging in modality-specific encoder networks, hinting at broader applications beyond video analysis. They also propose that the technique could benefit other healthcare/biomedical applications.

Summary updated

BibTeX
@inproceedings{rani2024activity,
  author = {Asha Rani and Yashaswi Verma},
  title = {{Activity-Based Early Autism Diagnosis Using a Multi-Dataset Supervised Contrastive Learning Approach}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure