Vision-Based Projection-Handwriting Integration in Classroom

Weihong Li, Hao Tang, Zhigang ZhuWorkshops @ CVPR 2006 · DOI

Introduction: The Challenge of the Virtual Classroom

In modern e-learning, the "record-and-playback" model often fails to capture the dynamic nature of a live lecture. Instructors frequently project digital slides onto a physical whiteboard and annotate them by hand. For remote students, this creates a disjointed experience: they see either a high-quality digital slide or a low-resolution video of the whiteboard, but rarely a seamless integration of both.

An instructor writing on a whiteboard over a projected slide, illustrating the registration challenge.

The primary technical hurdle is registration. Digital slides, ultrasonic handwriting trackers (like Mimio), and video cameras all operate in different coordinate systems with varying resolutions and optical distortions. Manual calibration—such as requiring an instructor to click the corners of a projection before every class—is a significant barrier to adoption. This research proposes a fully automated system that uses computer vision to align these disparate data streams into a single, high-resolution e-lecture format.

Methodology: The Video Bridge

The system utilizes a three-sensor setup: a PowerPoint logger, a Mimio Virtual Ink system for tracking pen movements via ultrasound, and a standard digital video camera. The core innovation lies in using the low-resolution video as a "bridge" to link the high-resolution digital slides with the physical handwriting data.

System overview showing the integration of handwriting (H), video (V), and slides (S) into a combined output.

The integration process involves two critical stages:

  1. Slide-to-Video Registration: The system first corrects radial distortion from the camera lens. It then employs a Laplacian-of-Gaussian (LoG) operator and a Hough Transform to detect the bright, rectangular boundary of the projected slide within the video frame. A 3x3 projective transformation matrix ($A_1$) is then calculated to map video pixels to the slide's coordinate system.
  2. Whiteboard-to-Video Registration: Because the whiteboard's physical frame is often invisible to the camera, the system matches content rather than boundaries. It "thins" handwriting from both the Mimio logs and the video frames into one-pixel-wide skeletons. Initial matching uses centroids and a "polar profile" to handle translation and scaling. Fine registration is achieved through adaptive template matching based on normalized cross-correlation, resulting in a second transformation matrix ($A_2$).

Key Results and Insights

The system was evaluated using a "Virtualized Classroom" setup at the City University of New York (CUNY), featuring real-world lecture captures.

  • High Precision: The system achieved integration with approximately one-pixel misalignment, making the fusion virtually imperceptible to viewers.
  • Efficiency: Calibration is remarkably fast, taking about 2 seconds on standard hardware (Pentium 1.2GHz), which supports near-real-time processing.
  • Robustness: By analyzing video frames immediately before or after a stroke is recorded, the system can find "clean" frames where the instructor does not occlude the camera's view of the board. It successfully integrates handwriting both inside and outside the projected slide area.

The most significant insight is that the low-resolution video, despite its quality, serves as the only sensor capable of "seeing" both the digital projection and the physical ink, acting as a Rosetta Stone for the entire system.

Conclusion and Future Directions

While the current system is highly effective, it assumes the camera and projector remain stationary after initial calibration. Future work aims to enhance the system's flexibility and intelligence:

  • Instructor Overlay: Using computer vision to "cut out" the instructor from the video and overlay them onto the digital slide, preserving non-verbal cues and gestures.
  • Automated Event Recognition: Implementing Bayesian networks to detect classroom events, such as board erasing, to automatically manage digital "pages."
  • Semantic Analysis: Moving from image-based capture to handwriting recognition, enabling text-based searchability within recorded lectures.

By automating the fusion of digital and physical media, this research paves the way for more immersive and accessible virtual classroom environments.

Summary updated

BibTeX
@inproceedings{li2006vision,
  author = {Weihong Li and Hao Tang and Zhigang Zhu},
  title = {{Vision-Based Projection-Handwriting Integration in Classroom}},
  booktitle = {Workshops @ CVPR},
  year = {2006},
  doi = {10.1109/cvprw.2006.212},
}
Figure