From Crayons to Code: AI-Driven Insights into a Child's Mental Health Through Drawings

Moomal Farhad, Mohammad Mehedy Masud, Aisha Alnaqbi, Rawan Mubarak, Aaisha Aladawi, Sara AlnaqbiAAAI 2025 · DOI

TL;DR

This paper introduces Siamese++, a novel AI framework for automatically screening children's mental health using their drawings, specifically from the Draw-A-Person (DAP) test. By leveraging an adaptive distance threshold, Siamese++ effectively handles the significant class imbalance in drawing datasets, achieving high accuracy and offering a scalable, interdisciplinary solution for early psychological intervention.

What Problem Does This Paper Address?

Childhood mental health is a critical but often overlooked area, with many children struggling to articulate their emotions. Early identification and intervention are crucial to prevent long-term psychological issues. Traditional methods for assessing children's mental state through drawings, like the Draw-A-Person (DAP) test, are subjective and time-consuming. This paper addresses the need for an automated, objective, and scalable solution to analyze children's drawings for psychological screening, bridging the gap between psychological expertise and AI.

Methods

The authors propose two main approaches for analyzing children's drawings:

  1. Feature Extraction:
    • Manual Feature Extraction: Experts identify key visual features in drawings (e.g., presence of body parts, use of color, placement) that are indicative of a child's mental state. These features are then manually extracted.
  2. LLM-Extracted Features: A novel technique using prompt engineering with state-of-the-art Large Language Models (LLMs) like GPT-4o is employed to automatically extract the same set of features from images. This demonstrates the potential of LLMs in interpreting complex visual data for diagnostic purposes.
  3. Image Classification:
    • Traditional ML Models: Machine learning classifiers (Random Forest, SVM, XGBoost) are trained on the extracted features.
  4. CNN-based Models: Convolutional Neural Networks (CNNs) are trained on raw image data.

A significant challenge in this domain is class imbalance, where drawings indicating a "need" for referral are far fewer than those indicating "no need." To address this, the paper introduces Siamese++, a novel Siamese network. Unlike traditional Siamese networks that rely on nearest-neighbor classification, Siamese++ uses an adaptive distance threshold. This threshold is calculated based on the mean distances between positive and negative class pairs, allowing for a more flexible and robust decision boundary, especially crucial when dealing with highly imbalanced datasets and intra-class variations.

Datasets

The primary dataset used is the "Draw-a-Person hand-drawn sketches" dataset from Kaggle, containing approximately 1,000 drawings from primary school students aged 4-11. These drawings were labeled by psychology experts into two categories: 'need' (for further psychological referral) and 'no need'. The authors also mention ongoing data collection from local schools.

Key Insights and Ideas

  • LLMs for Feature Extraction: The successful use of LLMs like GPT-4o for automatically extracting clinically relevant features from drawings is a significant advancement. This suggests LLMs can be powerful tools for automating complex feature engineering tasks in specialized domains.
  • Adaptive Thresholding for Imbalance: The core innovation, Siamese++, tackles class imbalance not by oversampling or undersampling, but by adaptively adjusting the classification threshold. This is crucial because drawings within the same class can vary significantly, making fixed thresholds less effective.
  • Interdisciplinary Integration: The paper highlights the power of combining AI (specifically computer vision and LLMs) with psychological expertise to address real-world mental health challenges.
  • Drawing as a Rich Data Source: The study reinforces the idea that children's drawings are a valuable and nuanced source of information about their emotional and psychological state, accessible even when verbal expression is limited.

Key Results

  • Siamese++ Performance: Siamese++ achieved a high F1 score of 0.87 on the image classification task, outperforming traditional Siamese networks (F1 score of 0.78) and transfer learning models (VGG16, ResNet, which showed 0.00 performance without augmentation).
  • LLM Feature Effectiveness: Machine learning classifiers trained on LLM-extracted features (with SMOTE) achieved competitive F1 scores (e.g., XGBoost at 0.85) compared to manually extracted features (XGBoost at 0.76), demonstrating the LLM's capability.
  • Robustness to Imbalance: Siamese++ achieved its high F1 score (0.87) even with a significant class imbalance (37 'need' vs. 915 'no need' drawings) and without image augmentation.
  • Parameter Sensitivity: The adaptive cutoff parameter 'α' in Siamese++ was found to be most effective around 0.5, indicating a balanced approach to incorporating intra-class and inter-class distances.

Limitations and Future Work

  • Dataset Size: While the Kaggle dataset is substantial, the authors acknowledge the ongoing process of collecting more data from local schools, suggesting that larger and more diverse datasets could further improve model performance.
  • Generalizability: The current study focuses on the DAP test. Future work could explore applying similar AI-driven analysis to other drawing-based psychological assessments (e.g., HTP test) or different age groups.
  • Clinical Validation: While expert labeling was used, further validation with clinical practitioners and longitudinal studies would be beneficial to confirm the real-world clinical utility of the proposed system.
  • LLM Interpretability: While LLMs were used for feature extraction, further research could explore the interpretability of how the LLM arrives at these features to gain deeper insights into the drawing analysis process.

Summary updated

BibTeX
@inproceedings{farhad2025from,
  author = {Moomal Farhad and Mohammad Mehedy Masud and Aisha Alnaqbi and Rawan Mubarak and Aaisha Aladawi and Sara Alnaqbi},
  title = {{From Crayons to Code: AI-Driven Insights into a Child's Mental Health Through Drawings}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35160},
}
Figure