Tangible mixed reality desktop for digital media management

Stefan Winkler, Hang Yu, Zhiying ZhouProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2007 · DOI

TL;DR

This paper introduces a tangible mixed-reality desktop system that uses computer vision to track hand and finger movements in 3D space, enabling intuitive gesture-based control for digital media management. By leveraging stereo cameras and triangulation, the system allows users to directly manipulate virtual objects without any wearable sensors, bridging the gap between physical and digital interactions.

What Problem Does This Paper Address?

Traditional human-computer interfaces, like keyboards and mice, are often inefficient and can feel disconnected from real-world interactions. While multi-modal interfaces have emerged, users often still need to adapt to specific devices and learn new interaction paradigms. The authors highlight that managing digital media, such as sorting photos, can be tedious compared to physical tasks like arranging photo prints. Furthermore, current computer interactions are largely solitary, hindering collaborative work. This paper aims to create a more natural, intuitive, and potentially collaborative interface for digital media management by allowing users to interact with virtual objects in 3D space using their bare hands.

Methods

The core of the system relies on computer vision techniques for gesture recognition. The process involves:

  • Hand Segmentation: This preliminary step uses image differencing to extract areas of interest containing hands. A running average of background frames is used to improve robustness to changing illumination.
  • Fingertip Detection: Fingertips are identified based on their circular border and the characteristic shape of a finger (a chain of non-filled pixels leading to a filled pixel region). The direction of the finger is inferred from the fingertip's position relative to a short chain of filled pixels.
  • Fingertip Classification and Tracking: Detected fingertips are classified based on their relationship to the hand. While the system re-computes fingertip locations for each frame rather than maintaining a persistent track, this approach offers stability for fast finger movements.
  • 3D Localization: Using a calibrated stereo camera setup, stereo triangulation is employed to determine the 3D coordinates of the detected fingertips. This extends the interaction space from a 2D plane to a 3D volume above the desktop.
  • Virtual Object Insertion: The system computes the 3D direction of the fingers to orient virtual objects. These virtual objects are then attached to the fingertips, allowing for direct manipulation in 3D space.

Datasets

Not discussed in the paper. The system appears to be demonstrated with real-time capture from stereo cameras.

Key Insights and Ideas

  • Bridging Physical and Digital: The system's primary insight is enabling a tangible interaction metaphor for digital media. By allowing users to "touch" and manipulate virtual objects with their fingers in 3D space, it mimics real-world interactions, making digital tasks more intuitive.
  • Unencumbered Interaction: A significant advantage is the elimination of wearable sensors. Relying solely on computer vision makes the system more accessible and user-friendly.
  • Extending Interaction Space: Moving beyond 2D tabletop interactions to a full 3D space above the surface provides users with greater freedom and a richer interaction environment.
  • Seamless Integration: The goal is to integrate virtual objects into the real world such that they appear to be part of the user's physical environment, enhancing immersion.

Key Results

The paper demonstrates the system's capability in performing several digital media management tasks:

  • Selecting an object: Pointing a finger at a virtual image selects it.
  • Dragging an object: Selected objects can be moved by moving the finger.
  • Resizing an object: A two-finger gesture controls the resizing of an object, with the separation between fingers dictating the magnitude of adjustment.
  • Deleting an object: Objects can be dragged into a physical "bin" (a paper box in the demonstration) to be deleted.

The system achieves consistent detection at 28 frames per second on a PC with a Pentium D processor, which is attributed partly to the absence of a complex tracking mechanism.

Limitations and Future Work

  • Robustness to Lighting and Background: The system's performance can be affected by ambient lighting conditions and the presence of objects similar in shape to fingertips. Dark objects or shadows can also interfere with hand segmentation. The authors suggest installing spot lights as a potential solution.
  • Palm vs. Back of Hand: The algorithm is more robust when the palm faces the camera, as fingernails can sometimes cause detection issues.
  • Occlusion Handling: While virtual objects are rendered in front by default, the authors acknowledge that more sophisticated rendering that accounts for occlusion by real objects (like fingers) could improve realism.
  • Future Work: The authors plan to extend the system to include additional interaction techniques and explore other applications beyond digital media management. They also mention the potential for multi-user support and collaborative manipulation.

Summary updated

Abstract

This paper presents a tangible mixed reality desktop that supports gesture-oriented interactions in 3D space. The system is based on computer vision techniques for hand and finger detection, without the need for attaching any devices to the user. The system consists of a pair of stereo cameras that point to a planar surface as the work bench. Using stereo triangulation, the 3D locations and directions of the user's fingers are detected and tracked in the space on and above the surface. Based on our 3D finger tracking technique, we design a few simple multi-finger gestural interactions for digital media management. The system provides a convenient and user-friendly way of manipulating virtual objects in 3D space and supports seamless interactions with physical objects.
BibTeX
@inproceedings{winkler2007tangible,
  author = {Stefan Winkler and Hang Yu and Zhiying Zhou},
  title = {{Tangible mixed reality desktop for digital media management}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2007},
  doi = {10.1117/12.703906},
}
Figure