Robotics via Experiential Active Learning With Immersive Technology: A Pedagogical Framework Designed for Engineering Curricula

Liangjing Yang, Songjie Xiao, Tengyue Wang, Yunze Shi, Katherine Driggs-CampbellIEEE Access 2024 · DOI

Introduction

Robotics is a multidisciplinary field requiring mastery of mechanics, electronics, and computer science. For engineering students, the transition from theoretical classroom concepts to the operation of physical industrial hardware is often jarring. Traditional laboratory settings frequently suffer from a "stiff prerequisite" barrier: limited access to expensive industrial robots and the reliance on bulky, shared "teach pendants" (handheld controllers) often results in a passive learning environment where only one student actively engages while others observe. This creates a bottleneck that hinders the development of practical skills and reduces student confidence.

The REALITy Framework

To address these pedagogical bottlenecks, the researchers developed the REALITy (Robotics via Experiential Active Learning with Immersive Technology) framework. This cyber-physical teaching platform integrates three distinct environments to facilitate a smoother learning curve:

  1. Virtual Environment: A digital twin of the laboratory space built in Unity3D.
  2. Mixed Reality (MR) Environment: Utilizing Microsoft HoloLens 2 headsets, students interact with virtual robot interfaces overlaid on the physical world.
  3. Physical Environment: An industrial UR3e robot arm controlled via the Robot Operating System (ROS2).
The Virtuality Continuum in the REALITy framework, bridging virtual interaction and physical reality.

The core innovation lies in the MR-based operating platform. By virtualizing the teach pendant, every student in a group gains a personalized, scalable holographic interface. This democratization of control allows students to program robot logic in a shared virtual space before validating movements on physical hardware, effectively turning a complex programming task into a collaborative spatial puzzle.

Curriculum Integration and System Flow

The REALITy framework is designed to map directly onto a standard robotics engineering syllabus, progressing from fundamentals to advanced topics. The system flow mirrors this pedagogical progression, ensuring that students understand the underlying mechanics before moving to complex control and perception tasks.

Integration of the robotics syllabus with the cyber-physical system flow across different learning stages.

As shown in the framework's architecture, the learning process is divided into three stages:

  • Fundamentals (Weeks 1-4): Focus on mechanics and modeling, including spatial representation and kinematics.
  • Essentials (Weeks 5-9): Focus on dynamics and control systems.
  • Advanced (Weeks 10-14): Focus on planning and perception, including robot vision and trajectory generation.

Experimental Evaluation

The framework was evaluated through a user study involving 18 engineering students (ages 20–23) tasked with solving a "Tower of Hanoi" pick-and-place operation. The study compared three modes: a purely physical Baseline, a purely Virtual (VR) environment, and the Mixed Reality (MR) REALITy framework. Evaluation metrics included NASA-TLX subjective workload scores, time-to-completion for each task step, and the frequency of troubleshooting required for a faultless automated flow.

Key Results

  • Workload Reduction: The MR group reported significantly lower mental and physical demand compared to the baseline physical setup. The virtualization of the interface solved the "passive observer" problem, fostering active participation for the entire group.
  • Accelerated Learning: While the MR group faced a higher "initial cost" in adaptation time (taking longer to complete the first step), their subsequent learning rate was more than double that of the baseline group. The researchers modeled this using an exponential decay function, finding a learning rate of 0.442 for MR versus 0.192 for the Baseline.
  • Enhanced Accuracy: Both VR and MR environments led to a significant reduction in troubleshooting steps, providing a safer and more effective space for debugging logic compared to direct physical interaction.

Discussion and Future Work

The study suggests that the ability to validate virtual code on a physical robot arm provides a critical "reality check" that bolsters student confidence and comprehension in a way that pure simulation cannot. However, hardware limitations remain, including eye fatigue from prolonged headset use and the need for longer adaptation periods for gesture-based controls.

Future iterations of the REALITy framework will focus on integrating "usage reminders" to prevent eye strain, optimizing the user interface to reduce initial adaptation costs, and expanding the curriculum to include more complex subjects like mechatronics and machine vision.

Summary updated

BibTeX
@article{yang2024robotics,
  author = {Liangjing Yang and Songjie Xiao and Tengyue Wang and Yunze Shi and Katherine Driggs-Campbell},
  title = {{Robotics via Experiential Active Learning With Immersive Technology: A Pedagogical Framework Designed for Engineering Curricula}},
  journal = {IEEE Access},
  year = {2024},
  doi = {10.1109/access.2024.3393473},
}
Figure