An IoT-Enabled Platform for the Assessment of Physical and Mental Activities Utilizing Augmented Reality Exergaming

Dionysios Koulouris, Andreas Menychtas, Ilias MaglogiannisSensors 2022 · DOI

TL;DR

This paper introduces an IoT-enabled platform that merges Augmented Reality (AR) exergames with wearable sensors to promote physical activity and assess user health. It offers a flexible, commodity-device-based solution for personalized healthcare and rehabilitation, allowing professionals to monitor and adapt game scenarios remotely.

What Problem Does This Paper Address?

The paper tackles the growing issue of sedentary lifestyles and the need for engaging, accessible tools to promote physical and mental well-being, particularly within healthcare and assisted living contexts. Traditional exergames often require expensive, specialized hardware, limiting their accessibility. Furthermore, there's a need for systems that can not only encourage activity but also provide objective, real-time health and cognitive assessments for healthcare professionals. The authors aim to bridge this gap by creating a platform that leverages ubiquitous mobile devices and IoT sensors for a more integrated and personalized approach to health-focused gamification.

Methods

The core of the proposed solution is a modular platform comprising three main components:

  • AR Game Development Framework (Mobile Application): This framework utilizes Google ARCore for AR functionality, enabling the display of virtual objects and challenges within the user's real-world environment. It handles game logic, user interaction, and the collection of basic sensor data (e.g., hand trembling, movement) from the mobile device. Game quests and parameters can be pre-configured or personalized by healthcare professionals.
  • Smartwatch Application: This component integrates with Wear OS smartwatches to capture and relay vital biosignal data, such as heart rate and step count, to the mobile application via Bluetooth.
  • Cloud Platform and Web Application: A serverless cloud backend stores game configurations, user results, and health data. A web application provides an administrative interface for healthcare professionals to customize game scenarios, monitor user progress, visualize health metrics, and analyze performance data.

The platform employs an anchor placement algorithm to ensure AR objects remain stable in the environment, considering factors like distance from the camera and other anchors.

Datasets

Not discussed in the paper. The evaluation was performed using the developed prototype on four different mobile devices (Galaxy A20, Galaxy A7 2018, TCL 20 5G, Huawei P20 Lite) and involved 18 users aged 20-40 and 15 users aged 40-60.

Key Insights and Ideas

  • Commodity Device Accessibility: By leveraging existing mobile phones and smartwatches, the platform significantly lowers the barrier to entry for AR-based health interventions, making them more accessible than systems requiring dedicated gaming consoles or specialized hardware.
  • Integrated Health Monitoring: The seamless integration of AR exergaming with IoT wearable sensors provides a holistic view of user engagement, physical activity, and physiological responses in real-time. This moves beyond simple activity tracking to a more comprehensive health assessment.
  • Personalized Healthcare Pathways: The ability for healthcare professionals to remotely configure game parameters, adapt challenges, and analyze detailed user data allows for truly personalized care plans and interventions, a crucial aspect for effective rehabilitation and chronic disease management.
  • Gamification for Motivation: The use of gamification elements, including scoring, leaderboards, and varied quest types, is shown to be effective in increasing user motivation for both physical activity and cognitive engagement.
  • Quantified Self for Clinical Insight: The system supports the "quantified self" movement by collecting and presenting detailed health data, which can be invaluable for diagnosing and monitoring conditions like tremor-based muscle or neural diseases.

Key Results

  • Performance: The AR exergame demonstrated playable frame rates (around 30 FPS) on tested commodity devices, with minor frame drops observed during extended sessions or in challenging lighting conditions. Battery drain was generally low, ranging from 0.6% to 2.8% per session.
  • User Engagement: The game was tested on 33 users across two age groups (20-40 and 40-60). Results indicated that users were physically active while playing, maintaining a good pace and average heart rate. The 40-60 age group showed slightly longer average game durations, more steps, and higher trembling scores compared to the younger group.
  • Technology Acceptance: A technology acceptance questionnaire yielded positive feedback, with average results indicating users found the AR game to be a good idea, user-friendly, and potentially beneficial for their health and activity levels.

Limitations and Future Work

  • Limitations:
    • The prototype is currently limited to Android devices supporting Google AR Services.
  • Performance can be affected by lighting conditions and the complexity of the environment.
  • Some usability aspects could be improved, particularly for use in smaller rooms or more complex environments.
  • Future Work:
    • Extend support to iOS devices.
  • Integrate affective computing frameworks to assess users' emotional status.
  • Explore more advanced AR and computer vision engines for richer interactions.
  • Enhance game challenges and scenarios in collaboration with healthcare professionals, enabling dynamic adaptation based on care plans and monitored results.
  • Implement multi-user operations, both in the same physical location and remotely.
  • Integrate existing recommendations for quantified-self applications.

Summary updated

Abstract

Augmented reality (AR) and Internet of Things (IoT) are among the core technological elements of modern information systems and applications in which advanced features for user interactivity and monitoring are required. These technologies are continuously improving and are available nowadays in all popular programming environments and platforms, allowing for their wide adoption in many different business and research applications. In the fields of healthcare and assisted living, AR is extensively applied in the development of exergames, facilitating the implementation of innovative gamification techniques, while IoT can effectively support the users' health monitoring aspects. In this work, we present a prototype platform for exergames that combines AR and IoT on commodity mobile devices for the development of serious games in the healthcare domain. The main objective of the solution was to promote the utilization of gamification techniques to boost the users' physical activities and to assist the regular assessment of their health and cognitive statuses through challenges and quests in the virtual and real world. With the integration of sensors and wearable devices by design, the platform has the capability of real-time monitoring the users' biosignals and activities during the game, collecting data for each session, which can be analyzed afterwards by healthcare professionals. The solution was validated in real world scenarios and the results were analyzed in order to further improve the performance and usability of the prototype.
BibTeX
@inproceedings{koulouris2022iot,
  author = {Dionysios Koulouris and Andreas Menychtas and Ilias Maglogiannis},
  title = {{An IoT-Enabled Platform for the Assessment of Physical and Mental Activities Utilizing Augmented Reality Exergaming}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22093181},
}
Figure