Modeling Users' Cognitive Performance Using Digital Pen Features

Alexander Prange, Daniel SonntagFrontiers Artif. Intell 2022 · DOI

TL;DR

This paper introduces a novel approach to assess cognitive performance using digital pen features, moving beyond traditional content analysis to focus on the how of drawing. By analyzing over 170 digital pen features, they achieve up to 87.5% accuracy in classifying cognitive test results as healthy or suspicious, paving the way for more objective and accessible neurocognitive assessments.

What Problem Does This Paper Address?

Traditional neurocognitive assessments, often conducted with pen and paper, suffer from several limitations. The collected data is typically monomodal (written form) and requires manual scoring, which is time-consuming and prone to subjective bias based on the clinician's expertise. Furthermore, this manual process hinders immediate feedback and objective reporting, crucial for timely interventions and personalized treatment plans. While digital pens offer a way to capture richer data, existing methods often still rely on analyzing the content of the drawings, requiring task-specific knowledge and limiting generalizability. This paper aims to bridge this gap by developing a system that can predict cognitive performance solely from the behavioral characteristics of pen input, independent of the specific test's content.

Methods

The core of this work lies in the extraction and analysis of a comprehensive set of digital pen features. The authors implemented and made publicly available over 170 such features, encompassing geometrical, spatial, temporal, and pressure characteristics of pen strokes. These features are derived directly from the raw pen input stream.

They then employed a two-pronged approach:

  1. Traditional Content Analysis: An interactive cognitive assessment tool was developed that records user input with a digital pen and automatically scores tests based on established medical scoring schemes (e.g., Mendez scoring for CDT). This automates manual scoring and provides real-time feedback to clinicians.
  2. Behavioral Analysis (Novel Approach): This is the paper's main contribution. Instead of analyzing the content, they use the extracted digital pen features as input for supervised machine learning (ML) models to predict cognitive performance. This approach aims for task-agnostic classification, focusing purely on sketching behavior.

They evaluated various ML classifiers, including SVMs, Deep Learning, Random Forests, and AdaBoost, on a dataset of cognitive assessments. An AutoML approach was also explored for optimizing classification performance.

Datasets

The study utilized a dataset collected from 40 participants (aged 65-85) performing three common neurocognitive assessments:

  • Clock Drawing Test (CDT)
  • Trail Making Test (TMT) (Parts A and B)
  • Rey-Osterrieth Complex Figure Test (ROCF)

This resulted in a total of 240 sketches. For the ML experiments, these sketches were labeled as either "healthy" or "suspicious" based on expert scoring and established clinical cutoffs. The paper does not introduce a new benchmark dataset but rather uses existing cognitive tests to create a novel dataset for their specific analysis.

Key Insights and Ideas

  • Focusing on "How" not "What": The most significant insight is that cognitive performance can be effectively predicted by analyzing how a person draws (e.g., speed, pressure, stroke variation) rather than what they draw. This behavioral analysis is more generalizable across different cognitive tests.
  • Digital Pen Features as Powerful Predictors: The extensive set of over 170 digital pen features proved highly effective, outperforming previous approaches on the tested cognitive assessments.
  • Task-Agnostic Classification is Feasible: The success of predicting cognitive status without relying on test-specific content analysis demonstrates the potential for broader applications of digital pen technology in cognitive assessment.
  • AdaBoost Shines for Binary Classification: Interestingly, AdaBoost emerged as a top performer for the binary classification task (healthy vs. suspicious), suggesting its robustness in this specific scenario, even when other ML methods performed less remarkably on stroke-level features.
  • Sketch-Level Features Outperform Stroke-Level: The study found that using features derived from the entire sketch generally yielded better results than those derived from individual strokes, highlighting the importance of holistic drawing behavior.

Key Results

  • The proposed approach achieved up to 87.5% accuracy in a binary classification task (healthy vs. suspicious) for cognitive tests.
  • Using a feature set of 176 digital pen features, the ML models outperformed previous approaches on the CDT, TMT, and ROCF.
  • The AdaBoost classifier achieved the highest accuracy of 87.5% on the sketch-based feature set.
  • Linear SVMs and Random Forests also showed strong performance, with accuracies around 85.4%.
  • Stroke-based feature analysis yielded significantly lower accuracies, generally around 61-65%, suggesting that holistic sketch characteristics are more informative for this task.
  • The interactive cognitive assessment tool, which provides real-time feedback and automated scoring, was successfully deployed in a geriatrics daycare clinic.

Limitations and Future Work

  • Stroke-Level Feature Limitations: The study noted that not all digital pen features are well-defined or informative at the stroke level (e.g., average lift duration). This contributed to the lower performance of stroke-based approaches.
  • Annotation Granularity: The current annotation is sketch-wise (healthy/suspicious for the entire sketch). Future work could explore more fine-grained annotation, potentially labeling individual strokes associated with specific errors.
  • AutoML Performance: While AutoML was explored, its performance on stroke-based features was only marginally better than standard ML methods and still lagged behind sketch-based approaches. This might be due to data size limitations for AutoML.
  • Generalizability to Other Domains: While the approach shows promise, further evaluation in other domains (e.g., education) is suggested.
  • Feature Importance and Explainability: Future work could delve deeper into understanding which specific features are most influential for prediction and how to further improve the explainability of the ML models for clinicians.
  • Real-time Adaptation: The authors envision using these insights for real-time adaptation of task difficulty and content in interactive systems based on a user's cognitive state.

Summary updated

BibTeX
@article{prange2022modeling,
  author = {Alexander Prange and Daniel Sonntag},
  title = {{Modeling Users' Cognitive Performance Using Digital Pen Features}},
  journal = {Frontiers Artif. Intell.},
  year = {2022},
  doi = {10.3389/frai.2022.787179},
}
Figure