Gesturing and Embodiment in Teaching: Investigating the Nonverbal ‎Behavior of Teachers in a Virtual Rehearsal Environment ‎

Roghayeh Barmaki, Charles E. HughesAAAI 2018 · DOI

Abstract

Nonverbal communication, including posture and gestures, plays a vital role in classroom dynamics and student engagement. However, novice teachers often struggle with "closed" body language that can signal defensiveness or a lack of confidence. This study investigates an automated system using Kinect sensors and machine learning to provide real-time visual and haptic feedback to teacher candidates within the TeachLivE virtual rehearsal environment. Results indicate that immediate feedback significantly reduces closed gestures and that haptic feedback is particularly well-received for its non-intrusive nature, fostering more open and effective teaching styles.

Introduction

In educational settings, a teacher’s nonverbal cues are often as impactful as their verbal instructions. "Closed" postures—such as crossing arms, clasping hands behind the back, or placing hands on hips—can inadvertently convey a lack of confidence or openness, negatively affecting student rapport. Novice teachers frequently lack awareness of these habits. Traditional methods of correcting nonverbal behavior involve manual review of recorded sessions by experts, a process that is resource-intensive and provides delayed feedback. To overcome these barriers, this research proposes an automated, real-time feedback loop integrated into a virtual classroom simulator.

Methods and System Design

The researchers developed a gesture recognition engine leveraging the Microsoft Kinect V2 sensor. The system focuses on five specific closed postures: arms folded, hands clasped in back, hands on hips, and hands clasped in the lower or upper front.

System workflow for gesture recording, annotation, and real-time feedback application.

The technical pipeline involves recording training clips, annotating gestures, and generating a detector database using the Visual Gesture Builder (VGB). The system employs an ensemble of 1,000 decision-tree classifiers optimized via the AdaBoost algorithm to ensure high accuracy. Feedback is delivered through two primary modalities:

  1. Visual Feedback: A monitor displays a "traffic light" signal that turns from green to red the moment a closed gesture is detected.
  2. Haptic Feedback: A Myo armband provides timed vibrations, allowing teachers to receive corrections without breaking eye contact with their virtual students.

The study involved 30 participants from a university College of Education for the visual feedback study, with a subset participating in a follow-up haptic feedback evaluation.

Results

The gesture detection engine demonstrated high reliability, achieving an average accuracy of 96.5% ± 2.1% across the targeted postures.

Quantitative and qualitative results: (a) Closed Gesture Percentage (CGP) by group and session; (b) Participant survey results regarding learning perception and system naturalness.

As shown in Figure 3(a), participants receiving feedback showed a marked reduction in their Closed Gesture Percentage (CGP) compared to their baseline sessions. Furthermore, a "carry-over" effect was observed in Group B, where participants maintained more open postures in Session 2 even after the feedback was removed.

Qualitative results (Figure 3(b)) highlight that participants found the feedback helpful for increasing self-awareness. While 62.5% of users found the visual feedback setting natural and non-intrusive, qualitative feedback for the haptic armband was even more enthusiastic, as it allowed for better immersion in the teaching task without requiring the teacher to look away from their "students."

Discussion and Conclusion

This research underscores the importance of immediacy in behavioral training. By providing real-time corrections, the system allows teacher candidates to adjust their stance mid-interaction, reinforcing positive habits through embodied learning. The use of off-the-shelf hardware like the Kinect and Myo armband suggests that sophisticated pedagogical tools can be implemented cost-effectively.

Future work will focus on expanding the sample size for haptic studies and developing the system into a comprehensive automated behavioral annotation tool for teacher assessment. The technology also holds promise for other fields requiring high-stakes interpersonal communication, such as law enforcement de-escalation training.

Summary updated

BibTeX
@inproceedings{barmaki2018gesturing,
  author = {Roghayeh Barmaki and Charles E. Hughes},
  title = {{Gesturing and Embodiment in Teaching: Investigating the Nonverbal ‎Behavior of Teachers in a Virtual Rehearsal Environment ‎}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11394},
}
Figure