Extended multitouch: recovering touch posture and differentiating users using a depth camera

Sundar Murugappan, Vinayak, Niklas Elmqvist, Karthik RamaniUIST 2012 · DOI

TL;DR

This paper introduces "extended multitouch" (EMT), a richer interaction modality that goes beyond simple touch point detection to include finger/hand posture and user identification. They achieve this using a single, low-cost depth camera (like a Kinect) mounted above a surface, enabling more intuitive and informative interactions on any display.

What Problem Does This Paper Address?

Traditional multitouch interfaces, while ubiquitous, are limited to detecting where a touch occurs on a surface. They lack the ability to understand the nuances of human interaction, such as the specific finger or hand posture being used, or even who is touching the surface. This limits the complexity and expressiveness of multitouch applications. Existing solutions often require extensive instrumentation of the surface or the user (e.g., gloves), which is impractical for widespread adoption. The authors aim to overcome these limitations by leveraging readily available depth cameras to infer richer interaction information from bare-handed touch on any surface.

Methods

The core of their approach involves using a depth camera to capture 3D information about the scene.

  • Touch Sensing: They define virtual volumes above the surface to detect when objects (fingers) enter a "touch input region." This involves analyzing depth data to identify pixels within a specific distance range from the surface. Connected component analysis is used to identify "touch blobs," and size thresholds are applied to filter out noise.
  • Tracking Touch Points: To support dynamic interactions like dragging, they use a nearest-neighbor approach to associate touch points across consecutive frames, intelligently discarding outlier frames for robustness.
  • Mapping to Display Space: A calibration process maps the 3D camera coordinates to the 2D display coordinates, similar to traditional touch screen calibration.
  • Extended Multitouch (EMT) Features:
    • Posture Recovery: They define additional virtual volumes for the "fingers-palm region" and "wrist region." By analyzing the depth data within these regions, they extract the center of gravity and principal directions to infer palm and wrist positions and orientation.
  • Handedness and Finger Identification: A two-dimensional hand model is constructed by combining the processed binary images. They then analyze geometric properties like angles and distances between touch points and the wrist/palm to deduce finger positions and identify hand poses (e.g., one-finger, two-finger with/without thumb, etc.). Handedness is determined by the thumb's signed angle relative to the orientation vector.
  • User Differentiation: The system leverages the visibility of user arms in the depth camera's view to heuristically differentiate users, assuming relatively stable positions around the tabletop.

Datasets

Not discussed in the paper. The authors conducted their own user studies to collect data for evaluation.

Key Insights and Ideas

  • Depth Cameras as a Universal Touch Interface: The paper strongly advocates for depth cameras as a low-cost, versatile solution for turning any surface into an interactive multitouch display, eliminating the need for embedded sensors.
  • Beyond Touch Points: The Power of Posture and Identity: The core insight is that understanding how a user touches (posture, handedness) and who is touching significantly enriches the interaction space, enabling more nuanced and personalized applications.
  • Leveraging Geometric Cues: The method cleverly uses geometric properties of the hand and fingers (angles, distances, principal components) derived from depth data to infer complex information like finger identification and handedness.
  • Thumb as a Critical Feature: The presence of the thumb is highlighted as crucial for accurate handedness and finger identification, particularly in multi-finger gestures.
  • Design Implications for Richer Interactions: The paper opens up possibilities for context-aware menus, complex gestures, and personalized user experiences that were previously difficult or impossible with standard multitouch.

Key Results

  • Touch Sensing Accuracy: The average spread for touch sensing was 8mm, with individual user spreads ranging from 4mm to 13mm. The average accuracy for hitting a virtual button was 18mm diameter.
  • Posture and Handedness Accuracy:
    • Overall handedness identification accuracy was 93.2%.
  • "Thumb" poses for two- and three-finger gestures achieved high accuracy (98% and 97%), while "no thumb" poses were less accurate (77% and 86%).
  • Overall finger identification accuracy was 97.7%.
  • User Differentiation: While not explicitly quantified, the system demonstrated the ability to differentiate users based on their arm positions.
  • Application Prototypes: The approach was successfully demonstrated in a scratchpad application and a more complex sketching tool called uSketch, showcasing its practical utility.

Limitations and Future Work

  • User Differentiation Limitations: The current user differentiation relies on users maintaining relative positions and not getting too close. It also doesn't identify specific users, only differentiates them.
  • Thumb Dependency: Accuracy for handedness and finger identification is significantly lower for poses lacking a thumb.
  • Ambiguous Poses: Some poses, particularly single-finger poses without a clear thumb indication, can be ambiguous.
  • Future Work:
    • Integrating a second vision-based tracking system for more robust participant tracking.
  • Improving hand model accuracy by reasoning with contours and convexity defects.
  • Exploring more complex gestural interactions and combining multiple input modalities for richer design spaces.
  • Further investigation into the impact of camera height on accuracy.

Summary updated

BibTeX
@inproceedings{murugappan2012extended,
  author = {Sundar Murugappan and Vinayak and Niklas Elmqvist and Karthik Ramani},
  title = {{Extended multitouch: recovering touch posture and differentiating users using a depth camera}},
  booktitle = {UIST},
  year = {2012},
  doi = {10.1145/2380116.2380177},
}
Figure