A Framework for Gesture Interface Design

Zhenyao Mo, Ulrich NeumannJ. Multim 2007

TL;DR

This paper introduces GeLex, a novel framework for designing gesture interfaces that simplifies gesture representation and recognition. By moving from low-level joint angles to a "lexical" space of finger poses and relations, it allows for the automatic generation of gesture interfaces from simple descriptions, making gesture interaction more accessible for developers and customizable for users.

What Problem Does This Paper Address?

Creating intuitive and effective gesture interfaces for applications, particularly in virtual environments and mobile settings, has been a significant challenge. Traditional approaches often involve hard-wiring gesture recognition directly into application design, making it time-consuming, complex, and inflexible for users. Furthermore, existing gesture recognition methods struggle with efficiency and robustness, while gesture notation systems are either too limited or require extensive learning. This paper aims to bridge this gap by providing a framework that decouples gesture interface design from application implementation, enabling rapid development and user customization.

Methods

The core of the proposed framework lies in two key components:

  • GeLex (Gesture Lexicon): A new, human-friendly notation system for describing hand poses. GeLex simplifies gesture representation by breaking down hand poses into a "finger-pose alphabet" and their inter-relations, rather than relying on complex 3D joint angles. This system allows for both full and partial hand pose descriptions, offering flexibility.
  • Generic Gesture Recognition Framework: The framework utilizes a generic hand pose recognition algorithm that can be adapted to specific applications through an interface description written in GeLex. The paper details a vision-based hand pose recognition algorithm that uses low-resolution depth images from a Canesta camera. This algorithm segments the hand, locates the palm and fingers, and then classifies finger poses and their inter-relations. The recognized hand pose is then encoded into a vector and compared against pre-defined template vectors derived from GeLex descriptions.

Datasets

Not discussed in the paper. The paper focuses on a general framework and a specific implementation using a Canesta depth-image camera. The "41 basic hand poses in SignWriting" are used as a benchmark for the expressiveness of GeLex, but not as a formal dataset for training or evaluation in the context of the proposed framework.

Key Insights and Ideas

  • Lexical Space for Gestures: The shift from high-dimensional joint angle space to a "lexical" space of finger poses and relations is a crucial insight. This abstraction simplifies both the description and recognition of gestures, making the process more manageable.
  • Decoupling Design and Implementation: The framework's ability to automatically generate a gesture interface from a description is a significant advancement. This separation allows designers to focus on the interface logic without getting bogged down in the intricacies of gesture recognition algorithms.
  • User Customization: The use of partial poses in GeLex and the vector-based matching mechanism allow for user customization of gesture interfaces, catering to individual preferences and comfort levels.
  • Efficiency with Depth Images: The use of low-resolution depth images from a Canesta camera, combined with a robust segmentation and recognition algorithm, enables real-time performance, a critical factor for interactive applications.

Key Results

  • Rapid Prototyping: The paper demonstrates the framework's effectiveness with a "Rock-Scissor-Paper" game example. The entire game was built in 30 minutes, with the gesture interface taking only 10 minutes to develop using a GeLex script of just three lines.
  • Real-time Performance: The hand pose recognition algorithm runs in real-time, consuming approximately 34% of the CPU on a Pentium IV 1.0GHz processor.
  • Expressive Notation: GeLex can represent all 41 basic hand poses from SignWriting, demonstrating its expressiveness.
  • Learnability: Users with no prior experience in computer science or gesture interfaces were able to learn GeLex in under 30 minutes.

Limitations and Future Work

  • Static Hand Postures: The current implementation is limited to recognizing static hand postures. The authors acknowledge that their final goal is to include dynamic gestures.
  • Frontal View Constraint: The generic pose recognition algorithm is limited to frontal views within a +/-30-degree rotation due to the low-resolution depth images.
  • Distinguishable Poses: The algorithm may struggle to distinguish between certain hand poses in low-resolution images, and poses that cannot be represented by the GeLex schema are also a limitation. Motion blur can also lead to recognition failures.

Future work will focus on incorporating dynamic gestures and improving robustness to different viewing angles and image conditions.

Summary updated

BibTeX
@article{mo2007framework,
  author = {Zhenyao Mo and Ulrich Neumann},
  title = {{A Framework for Gesture Interface Design}},
  journal = {J. Multim.},
  year = {2007},
}
Figure