Investigating Visual Perception Impairments through Serious Games and Eye Tracking to Anticipate Handwriting Difficulties

Chiara Piazzalunga, Linda Greta Dui, Cristiano Termine, Marisa Bortolozzo, Matteo Matteucci, Simona Ferrante

TL;DR

This paper introduces a novel approach to early dysgraphia screening by using a gamified visual perception test combined with eye-tracking. Machine learning models, particularly those leveraging game performance data, show promising results in predicting children at risk of dysgraphia, suggesting a more accessible and less stressful diagnostic tool.

What Problem Does This Paper Address?

Dysgraphia, a learning disability affecting handwriting, is often diagnosed late, after handwriting development is complete. This delay hinders timely intervention and can lead to significant psychological distress for affected children. Current diagnostic methods can be subjective, time-consuming, and administered when the problem is already established. The paper addresses the need for an earlier, more objective, and less stressful screening method by investigating the link between visual perception skills and handwriting difficulties. Specifically, it explores whether gamified visual perception tasks, coupled with eye-tracking data, can serve as early indicators of dysgraphia.

Methods

The researchers developed a 2D, single-player serious game designed for primary school children. This game incorporates gamified versions of standard clinical visual perception tests (e.g., form constancy, masked form constancy, copying, tracing, figure-ground perception, visual closure) across three difficulty levels. Participants played these games while wearing an eye-tracker to record their gaze patterns and fixations.

To predict dysgraphia risk, five machine learning models (Gaussian Naive Bayes, Random Forest, Support Vector Machine, CatBoost) were trained and evaluated. The models used combinations of features derived from:

  • Game Performance: Time to completion, errors.
  • Eye-Tracking Data: Gaze positions, fixations, indecisiveness (number of gaze changes between elements), attention (percentage of gaze on target).
  • Drawing Data: Features related to tract continuity, length, and spatial organization from copying and tracing tasks.

A Nested Cross-Validation approach was employed to ensure robust evaluation on a small dataset. SHAP (SHapley Additive exPlanations) was used to interpret the models and identify the most influential features.

Datasets

The study utilized a dataset of 53 primary school children (aged 7.84 ± 0.90).

  • Training/Testing Data: This dataset was derived from the children's performance in the serious game (game performance, eye-tracking, and drawing data) and their scores on the BVSCO-2 (Batteria per la Valutazione della Scrittura e della Competenza Ortografica) writing speed test. The BVSCO-2 results were dichotomized into "at-risk" and "not-at-risk" categories to serve as the ground truth for the machine learning models.
  • Normative Data: The BVSCO-2 test itself relies on normative data obtained from over 350 children.

No new benchmark datasets were introduced; rather, existing clinical assessments were adapted and integrated into a novel gamified framework.

Key Insights and Ideas

  • Visual Perception as a Predictor: The study strongly supports the hypothesis that visual perception abilities are significantly linked to handwriting performance and can serve as an early indicator of dysgraphia.
  • Gamification for Engagement: Using serious games for assessment proved highly effective, with a reported usability score of 89.4 ± 9.6, indicating excellent acceptance and engagement among children. This suggests gamification can mitigate the stress and anxiety often associated with traditional testing.
  • Eye-Tracking for Subtle Clues: Eye-tracking data, particularly measures of indecisiveness and attention, provided valuable insights into cognitive processes underlying visual perception and their relation to handwriting risk.
  • Game Performance Dominance: Interestingly, game performance metrics alone (time and errors) emerged as the strongest predictors of dysgraphia risk, achieving an F1 score of 0.71 on the test set. This highlights the potential of a game-based assessment without the need for more complex equipment like eye-trackers.
  • Drawing Features Matter: While game performance was strongest, drawing features, especially those related to tract discontinuity and spatial organization in copying tasks, also contributed significantly to predicting risk.
  • Holistic Approach: The most effective models incorporated a combination of features, suggesting that visual perception, eye movements, and drawing skills collectively contribute to handwriting proficiency.

Key Results

  • Machine Learning Performance: The CatBoost classifier, using game performance data, achieved an F1 score of 0.77 on the training set and 0.71 on the test set, demonstrating good predictive power for dysgraphia risk.
  • Usability: The serious game achieved an excellent usability score of 89.4 ± 9.6, indicating high acceptance by the child participants.
  • Feature Importance: SHAP analysis revealed that Visual Closure, Masked Form Constancy, and Figure-Ground Perception games were particularly influential in predicting risk when using game performance data. For eye-tracking, indecisiveness (number of gaze changes) and average fixation duration were key indicators. For drawing, tract discontinuity and spatial organization were significant.
  • Gender Differences: A slight but statistically significant difference was observed, with males showing slightly worse performances than females in the BVSCO-2, aligning with historical observations.

Limitations and Future Work

  • Sample Size: The study acknowledges a relatively small sample size (n=53), which can limit the generalizability of the findings. Future work should extend the research to a larger and more diverse population.
  • Eye-Tracker Challenges: The head-mounted eye-tracker presented some usability challenges, including discomfort and potential for chaotic data due to head movements. Future research could explore display-mounted eye trackers or alternative eye-tracking solutions.
  • Apple Pencil Data: The study did not fully integrate Apple Pencil data (e.g., pressure, tremor) into the predictive models, despite acknowledging its potential influence on handwriting. Future work could incorporate these quantitative measurements.
  • Narrowing Scope: The authors suggest that future research could benefit from narrowing the focus to specific visual perception abilities that demonstrated stronger predictive power.
  • Cost of Technology: While the game performance model showed strong results, the reliance on eye-trackers and iPads might still be a barrier for widespread adoption in schools. Future work could focus on developing standalone game-based screening tools that are more accessible.

Summary updated

Abstract

Dysgraphia is a learning disability that causes handwritten production below expectations. Its diagnosis is delayed until the completion of handwriting development. To allow a preventive training program, abilities not directly related to handwriting should be evaluated, and one of them is visual perception. To investigate the role of visual perception in handwriting skills, we gamified standard clinical visual perception tests to be played while wearing an eye tracker at three difficulty levels. Then, we identified children at risk of dysgraphia through the means of a handwriting speed test. Five machine learning models were constructed to predict if the child was at risk, using the CatBoost algorithm with Nested Cross-Validation, with combinations of game performance, eye-tracking, and drawing data as predictors. A total of 53 children participated in the study. The machine learning models obtained good results, particularly with game performances as predictors (F1 score: 0.77 train, 0.71 test). SHAP explainer was used to identify the most impactful features. The game reached an excellent usability score (89.4 ± 9.6). These results are promising to suggest a new tool for dysgraphia early screening based on visual perception skills.
BibTeX
@inproceedings{piazzalunga2023investigating,
  author = {Chiara Piazzalunga and Linda Greta Dui and Cristiano Termine and Marisa Bortolozzo and Matteo Matteucci and Simona Ferrante},
  title = {{Investigating Visual Perception Impairments through Serious Games and Eye Tracking to Anticipate Handwriting Difficulties}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=EiKcYMt1GA},
}
Figure