Augmented Reality Analytics to Investigate Motor Skills for Crossing the Midline

Manjeet Singh, Shaun Bangay, Atul SajjanharICCE 2021 · DOI

Abstract

Assessing "crossing the midline"—the ability to move a limb across the body's center—is vital for monitoring early childhood development. Traditional methods rely on subjective clinical observation. This paper introduces a prototype Augmented Reality (AR) drumming application that gamifies motor skill exercises while logging high-frequency spatial telemetry. By transforming physical interactions into visualizable data, the system offers a precise, quantifiable tool for therapists to evaluate motor control and developmental progress.

Introduction

In early childhood development, "crossing the midline" is a prerequisite for complex bilateral activities such as reading, writing, and sports. While clinical experts currently evaluate this skill through qualitative observation, such methods lack granular, objective data. Previous digital interventions using Virtual Reality (VR) faced limitations in immersion and physical context. This research proposes an AR-based approach to provide "quantifiable measurements to complex immersive interactions" while maintaining high levels of child engagement through play.

The Drum AR App

The researchers developed the Drum AR App using Unity3D and the Vuforia computer vision engine. The application operates on mobile devices (tablets or phones) and utilizes physical AR markers to anchor virtual objects in the real world.

The Drum AR App interface showing virtual mushrooms (drums) and a drumstick controlled via physical markers.

The core interaction involves virtual 3D mushrooms acting as drums. Audio instructions prompt the child to hit specific colored mushrooms. To hit a target on the opposite side of their dominant hand, the child must physically cross their body's midline. As the child interacts, the system logs telemetry data—including stick positions, hit timings, and marker events—directly into a JSON format.

Analytics Pipeline and Metrics

To make the raw telemetry actionable, the team developed an Analytics Plots Dashboard using Python and Streamlit. This dashboard parses the spatial data to generate 2D distance plots and 3D scatter plots of the drumstick's trajectory.

The Analytics Plots Dashboard visualizing a single move through a 2D distance plot and a 3D scatter plot of the drumstick trajectory.

The system captures a wide array of multi-dimensional metrics, ranging from engagement (correct/wrong hits) to motor control (hit rate timing and stick orientation). A key insight of this approach is the geometric representation of clinical success: a successful midline crossing is mathematically identified when the drumstick's path and the target drum's coordinates converge on a distance-vs-time plot.

Results and Discussion

The pilot study serves as a successful proof-of-concept for real-time AR logging. The system successfully captured high-frequency spatial data and transformed raw coordinates into intuitive visualizations. By framing a clinical test as a drumming game, the researchers aim to mitigate the boredom associated with conventional assessments, potentially leading to more natural and accurate performance data.

Conclusion and Future Work

While the current prototype validates the technical feasibility of AR analytics for motor skills, it is limited by hardware constraints requiring the child to hold the mobile device. Future iterations will explore head-mounted displays to enable two-handed drumming patterns. Subsequent research will also involve classroom testing to validate AR-derived metrics against traditional therapist observations, ensuring the digital data aligns with clinical reality.

Summary updated

BibTeX
@inproceedings{singh2021augmented,
  author = {Manjeet Singh and Shaun Bangay and Atul Sajjanhar},
  title = {{Augmented Reality Analytics to Investigate Motor Skills for Crossing the Midline}},
  booktitle = {ICCE},
  year = {2021},
  doi = {10.58459/icce.2021.4316},
}
Figure