Restoring Eye Contact to the Virtual Classroom with Machine Learning

Ross Greer, Shlomo Dubnov

TL;DR

Researchers at UC San Diego have developed a real-time machine learning pipeline that identifies which participant a user is looking at during a video call and overlays their name on the video feed. By transforming gaze estimation into a "target-oriented" prediction, the system restores the non-verbal cues essential for "semantic synchronization" in virtual music classrooms and ensembles.

What Problem Does This Paper Address?

In a physical classroom, a conductor can cue a musician with a simple glance. In a virtual classroom (like Zoom), this is impossible due to the "parallax" problem: if you look at a student’s video tile on your screen, you appear to be looking down or away to everyone else. If you look directly at the camera to simulate eye contact, you can’t actually see the students' reactions.

This loss of non-verbal communication is particularly devastating for music education. Musicians rely on eye contact to coordinate breathing, synchronize bow placement, and interpret improvisational cues. The authors seek to bridge this gap by creating a system that can "read" a user's gaze and explicitly communicate the intended target to the rest of the group.

Methods

The authors propose a modular, calibration-free pipeline designed for speed and real-time performance (targeting 30 fps).

  1. Face Detection: The system captures a 1920x1080 frame, resizes it, and uses a Single-Shot Detector (SSD) with a ResNet-10 backbone to find the user's face.
  2. Feature Extraction & Regression: The cropped face image is fed into a modified AlexNet. Crucially, the authors don't just use the image; they concatenate the facial bounding box coordinates (position and size) into the first fully connected layer. This provides the network with "coarse" spatial context about where the head is relative to the camera.
  3. Gaze Point Estimation: The network regresses to a 2D coordinate $(x, y)$ representing the gaze point on the monitor in centimeters relative to the camera.
  4. Target Mapping: The system captures the current screen layout (e.g., the grid of Zoom tiles) and uses a simple "nearest neighbor" logic. If the predicted gaze point is within a certain threshold of a specific participant's tile, that participant is identified as the target.
  5. Augmentation: Using Open Broadcaster Software (OBS), the system overlays the target's name onto the user's outgoing video feed, effectively "announcing" the gaze to the ensemble.

Datasets

The authors collected a custom dataset to train their model, as existing datasets (like MPIIGaze) didn't perfectly match the "user seated at a desk" constraints of a virtual classroom.

  • The Dataset: 161,634 image frames from five participants.
  • Ground Truth: Participants looked at 91 defined gaze locations across a 27-inch monitor.
  • Split: 129,103 frames for training, ~24k for validation, and ~8k for testing.

Key Insights and Ideas

  • Target-Oriented vs. Coordinate-Oriented: The authors realized that for videoconferencing, pixel-perfect accuracy isn't necessary. The system only needs to be accurate enough to distinguish between discrete "tiles" or zones. This allows for a more lightweight, faster model.
  • Horizontal Superiority: The study found that the model is significantly better at discriminating gaze along the x-axis (horizontal) than the y-axis. Consequently, the system performs much better when video tiles are arranged in a single horizontal row rather than a standard grid.
  • Context Matters: By feeding the bounding box data (face position/size) into the neural network alongside the image, the model learns to account for head pose and distance without needing a complex 3D head-pose estimation block.

Key Results

  • Speed: The system achieved an average inference time of ~0.032 seconds, or 30.9 fps, meeting the benchmark for smooth real-time video.
  • Accuracy: The Mean Squared Error (MSE) was 1.749 cm. In practical terms, the "hit rate" (correctly identifying the right Zoom tile) was as high as 98% for two users and remained above 88% for a horizontal layout of eight users.
  • Educational Impact: In a pilot study with 72 students performing "Soundpainting" (a gestural music language), 87% of students reported that the system helped establish a communicative relationship with the conductor.

Limitations and Future Work

  • Dataset Diversity: The current model was trained on only five subjects. The authors acknowledge the need to expand the dataset to include a wider variety of skin tones, facial features, and head poses to ensure the system is equitable and robust.
  • Integration: The current setup is a bit "hacky," requiring OBS and a local server to loop the video back into Zoom. Future work aims to develop this as a native plugin for conferencing software.
  • Privacy: The authors highlight the ethical need for a "kill switch," allowing users to opt-out of gaze tracking at any time to maintain privacy.

Summary updated

Abstract

Nonverbal communication, in particular eye contact, is a critical element of the music classroom, shown to keep students on task, coordinate musical flow, and communicate improvisational ideas. Unfortunately, this nonverbal aspect to performance and pedagogy is lost in the virtual classroom. In this paper, we propose a machine learning system which uses single instance, single camera image frames as input to estimate the gaze target of a user seated in front of their computer, augmenting the user’s video feed with a display of the estimated gaze target and thereby restoring nonverbal communication of directed gaze. The proposed estimation system consists of modular machine learning blocks, leading to a target-oriented (rather than coordinate-oriented) gaze prediction. We instantiate one such example of the complete system to run a pilot study in a virtual music classroom over Zoom software. Inference time and accuracy meet benchmarks for videoconferencing applications, and quantitati
BibTeX
@inproceedings{greer2021restoring,
  author = {Ross Greer and Shlomo Dubnov},
  title = {{Restoring Eye Contact to the Virtual Classroom with Machine Learning}},
  booktitle = {CSEDU},
  year = {2021},
  url = {https://openreview.net/forum?id=jG7u4YaesS},
}

Loading related content...

Figure