Interaction Between Real and Virtual Humans: Playing Checkers

Rémy Torre, Pascal Fua, Selim Balcisoy, Michal Ponder, Daniel ThalmannEGVE 2000 · DOI

TL;DR

This paper presents a real-time augmented reality system that allows a virtual human to interact with a real human playing checkers. By leveraging purely image-based techniques for tracking and registration, the system achieves seamless integration of virtual characters into the real world without cumbersome motion capture hardware.

What Problem Does This Paper Address?

The increasing demand for Augmented Reality (AR) systems necessitates the seamless integration of virtual humans into real-world environments, enabling realistic interactions between virtual and real entities. While virtual humans have been successfully integrated into purely synthetic virtual environments, their presence and interaction within AR settings, where they must perceive and react to real-world events and humans, remains a significant challenge. Specifically, this paper addresses the problem of enabling a virtual human to perceive and react to a real human's actions in real-time within an AR context, without requiring the real human to wear any special tracking equipment. The authors focus on the specific scenario of a checkers game to demonstrate their approach.

Methods

The core of the system relies on a Model-Based Tracking approach to achieve robust registration between the real and virtual worlds. This involves:

  • Camera Calibration and Motion Tracking: The system first calibrates the camera and then tracks its motion. This is achieved by using a 3D wireframe model of the checkers board. An objective function is minimized to automatically adjust the model's state variables and, if necessary, the camera's pose parameters, ensuring the model's projection aligns with the board in the image.
  • Motion Prediction: A linear motion predictor is used to estimate the 3D position of the model in the next frame, enhancing tracking robustness.
  • Iterative Minimization: A non-linear energy function is iteratively minimized with respect to the six 3D pose parameters of the camera relative to the 3D model. This process uses image gradients to fit the model to the observed scene.
  • Checkers Detection: To enable the virtual human to understand the game state, a checkers detection algorithm is employed. This algorithm works on the horizontal plane of the board and uses pixel values within projected scan segments to determine the presence or absence of a piece on a square.
  • Virtual Human Behavior: The virtual player's behavior is driven by a simple algorithm that takes the positions of real pieces as input and determines the virtual human's next move. The system also includes mechanisms to detect illegal moves.
  • Image Compositing: Chroma-keying (using black as the key color) is used to blend the rendered virtual scene with the real filmed scene, creating the augmented reality illusion.

Datasets

Not discussed in the paper. The system is demonstrated using a real-time interaction scenario with a physical checkers board and a single color camera.

Key Insights and Ideas

  • Leveraging the Environment for Registration: The paper demonstrates that environmental elements, like a checkers board, can serve as effective calibration objects for AR registration, eliminating the need for ad-hoc targets.
  • Robustness to Occlusion: The model-based tracking approach, which integrates information over the entire wireframe model, proves to be relatively insensitive to partial occlusions, a common issue in AR.
  • Enabling Non-Verbal Interaction: By allowing the virtual human to perceive the real human's actions through vision-based techniques, the system moves beyond simple AR overlays to enable more dynamic and interactive experiences.
  • Real-time Performance without Cumbersome Hardware: The achievement of real-time interaction without requiring the real user to wear motion capture suits or VR helmets is a significant step towards more natural and accessible AR applications.

Key Results

  • The tracking algorithm achieves a rate of 22 Hz on an SGI Octane workstation, enabling real-time performance.
  • The system demonstrates the ability to track the checkers board accurately even when the camera is rotating, translating, or changing focal length.
  • The virtual player can successfully interact with the real player in a checkers game, with the virtual human reacting to the real player's moves.
  • The image compositing technique effectively blends the virtual and real elements, creating a convincing augmented reality experience.

Limitations and Future Work

  • Initialization: The tracking algorithm is initialized semi-automatically using a mouse to supply an approximate initial position.
  • Checkers Detection Simplification: The checkers detection currently assumes pieces are only on squares of the same color and relies on a simple thresholding mechanism.
  • Virtual Human Grasping: The grasping action for the virtual human is described as a three-step subtask (reaching, grasping, animating), suggesting that complex object manipulation is handled through pre-defined animations.
  • Future Work: The authors plan to extend the approach to more generic situations and to simultaneously track multiple objects to improve robustness and precision. They are also working on a multi-threaded implementation for faster performance.

Summary updated

BibTeX
@inproceedings{torre2000interaction,
  author = {Rémy Torre and Pascal Fua and Selim Balcisoy and Michal Ponder and Daniel Thalmann},
  title = {{Interaction Between Real and Virtual Humans: Playing Checkers}},
  booktitle = {EGVE},
  year = {2000},
  doi = {10.2312/egve/egve00/023-032},
}
Figure