Interaction for Entertainment Contents Based on Direct Manipulation with Bare Hands

Kenji Oka, Imari Sato, Yasuto Nakanishi, Yoichi Sato, Hideki KoikeIFIP 2003 · DOI · OpenAlex

Abstract

This paper presents EnhancedDesk, an augmented reality tabletop system designed for natural and intuitive interaction with digital and physical objects using bare hands. By utilizing infrared thermal tracking, the system overcomes the limitations of traditional computer vision techniques in environments with dynamic projected lighting. We describe the hardware configuration, the fingertip tracking algorithm, and insights gained from a large-scale public deployment at SIGGRAPH 2001 involving over 2,000 participants.

Introduction

The development of "natural and intuitive" augmented desk interfaces has long been a goal in human-computer interaction. Early systems, such as Wellner’s DigitalDesk, established the foundation for tabletop AR but often relied on fragile computer vision techniques like color segmentation or background subtraction. In a typical augmented desk environment, a projector continuously modifies the surface's appearance. This dynamic lighting makes traditional camera-based tracking unreliable, as the perceived "color" of a user's skin and the desk background change with every frame of the projected image. EnhancedDesk addresses these challenges by employing thermal imaging to achieve robust, real-time tracking of multiple hands and fingertips.

System Architecture

The EnhancedDesk system utilizes a hardware-software hybrid approach to ensure stability under varying lighting conditions. The physical setup includes a plasma display, a half-transparent tabletop, and a dual-projector system (front and rear projection) to minimize shadows cast by the user's hands.

The EnhancedDesk hardware configuration, featuring infrared and color cameras alongside front and rear LCD projectors.

As shown in Figure 1, the system integrates an infrared camera specifically tuned to the range of human body temperature (30–34°C). This allows the system to binarize the hand region effectively, as the skin appears significantly brighter than the desk surface, regardless of the visible light being projected.

Tracking Methodology

The core of the EnhancedDesk's interaction model is its robust fingertip detection and temporal tracking algorithm.

  1. Thermal Binarization: The infrared feed is processed to isolate the hand from the background based on heat signatures.
  2. Fingertip Detection: The system identifies fingertips by searching for semi-circular patterns within the detected hand region using normalized correlation with a circular template.
  3. Temporal Tracking: To maintain identity across frames, a filtering technique predicts future finger positions and matches them to new detections.
Visualization of the real-time fingertip tracking system identifying three distinct points of contact.

This approach allows for the simultaneous tracking of multiple fingertips (Figure 2), enabling complex multi-touch gestures such as stretching virtual graphics or manipulating digital objects with both hands.

Evaluation and Public Deployment

The system underwent extensive real-world testing during a six-day exhibit at the Emerging Technologies venue of SIGGRAPH 2001. Over 2,000 visitors interacted with the desk, providing a diverse dataset of hand sizes, movement styles, and cultural backgrounds. The system demonstrated remarkable reliability, operating continuously without failure throughout the event. Qualitative feedback indicated that users found direct hand manipulation significantly more natural than traditional input devices like a mouse.

Discussion and Key Insights

The deployment revealed several critical insights into the design of bare-hand AR interfaces:

  • The Haptic Feedback Gap: User experience varied significantly based on the application type. In the StickyFace application (which involves distorting a virtual face), users were highly engaged despite the lack of physical touch. Conversely, in physics-based games like Rush&Crash, the absence of haptic resistance from virtual objects was perceived as "awkward," suggesting that haptic feedback is essential for realistic physics simulations.
  • Cultural Gesture Variance: The researchers discovered that gestures often assumed to be universal are culturally dependent. For instance, the "squeezing" gesture varies in how the thumb is positioned relative to the palm across different cultures, which initially led to errors in gesture recognition.
  • The "Cold Hand" Limitation: Thermal tracking's reliance on temperature differentials means that users with lower peripheral skin temperatures (noted more frequently in female participants) were sometimes difficult to track against the background.

Conclusion

EnhancedDesk demonstrates that thermal-based computer vision is a robust solution for bare-hand interaction in projected AR environments. While the system successfully tracks multiple fingertips in real-time, future work must address the integration of haptic feedback and the refinement of gesture recognition to account for cultural diversity and varying physiological signatures.

Summary updated

BibTeX
@inproceedings{oka2003interaction,
  author = {Kenji Oka and Imari Sato and Yasuto Nakanishi and Yoichi Sato and Hideki Koike},
  title = {{Interaction for Entertainment Contents Based on Direct Manipulation with Bare Hands}},
  booktitle = {IFIP},
  year = {2003},
  doi = {10.1007/978-0-387-35660-0\_48},
}
Figure