Abstract
Nonverbal communication, in particular eye contact, is a critical element of the music classroom, shown to keep students on task, coordinate musical flow, and communicate improvisational ideas. Unfortunately, this nonverbal aspect to performance and pedagogy is lost in the virtual classroom. In this paper, we propose a machine learning system which uses single instance, single camera image frames as input to estimate the gaze target of a user seated in front of their computer, augmenting the user's video feed with a display of the estimated gaze target and thereby restoring nonverbal communication of directed gaze. The proposed estimation system consists of modular machine learning blocks, leading to a target-oriented (rather than coordinate-oriented) gaze prediction. We instantiate one such example of the complete system to run a pilot study in a virtual music classroom over Zoom software. Inference time and accuracy meet benchmarks for videoconferencing applications, and quantitative and qualitative results of pilot experiments include improved success of cue interpretation and student-reported formation of collaborative, communicative relationships between conductor and musician.
Introduction
In a physical classroom, a conductor can cue a musician with a simple glance. In a virtual classroom environment, such as Zoom, this is hindered by the "parallax" problem: looking at a student’s video tile on a monitor makes the user appear to be looking down or away to other participants. Conversely, looking directly at the camera to simulate eye contact prevents the user from seeing the students' reactions.
This loss of non-verbal communication is particularly devastating for music education, where musicians rely on eye contact to coordinate breathing, synchronize bow placement, and interpret improvisational cues. We propose a system that "reads" a user's gaze and explicitly communicates the intended target to the rest of the group by overlaying the target's name on the video feed.
Methodology
The proposed system utilizes a modular, calibration-free pipeline designed for real-time performance, targeting 30 frames per second (fps).
System Architecture
The pipeline is divided into several functional blocks. First, a Single-Shot Detector (SSD) with a ResNet-10 backbone identifies the user's face within a 1920x1080 frame. The cropped face image is then processed by a modified AlexNet. To provide spatial context, the facial bounding box coordinates (position and size) are concatenated into the first fully connected layer, allowing the model to account for head pose and distance without a dedicated 3D estimation block.
The network regresses to a 2D coordinate $(x, y)$ representing the gaze point on the monitor. Finally, a target mapping system uses "nearest neighbor" logic to identify which participant tile the user is looking at based on the current screen layout.
Data Collection
A custom dataset was collected to match the "user seated at a desk" constraints. It consists of 161,634 image frames from five participants looking at 91 defined gaze locations across a 27-inch monitor. The data was split into training (129,103 frames), validation (~24k), and testing (~8k) sets.
Results and Discussion
The system achieved an average inference time of approximately 0.032 seconds (30.9 fps), meeting the requirements for smooth real-time video. The Mean Squared Error (MSE) for gaze point estimation was 1.749 cm.
In practical application, the "hit rate" for correctly identifying a Zoom tile was as high as 98% for two users and remained above 88% for a horizontal layout of eight users. A key insight was "Horizontal Superiority": the model is significantly better at discriminating gaze along the x-axis than the y-axis, suggesting that horizontal tile arrangements are optimal for this technology.
In a pilot study with 72 students performing "Soundpainting," 87% of participants reported that the system helped establish a communicative relationship with the conductor, confirming the educational value of restored eye contact.
Conclusion
While successful, the current model's reliance on a small training group necessitates future expansion to include more diverse skin tones and facial features. Additionally, moving from an external setup (using OBS) to a native plugin would improve accessibility. Ethical considerations, such as a privacy "kill switch," remain paramount as this technology moves toward broader integration in virtual collaboration tools.
Summary updated