Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor

David Kim, Otmar Hilliges, Shahram Izadi, Alex Butler, Jiawen Chen, Iason Oikonomidis, Patrick OlivierUIST 2012 · DOI

TL;DR

This paper introduces "Digits," a novel wrist-worn sensor system that enables full 3D hand tracking for mobile, freehand interactions without requiring gloves or external sensors. By combining IR laser line projection and diffuse IR LEDs with advanced kinematic models, Digits achieves a compelling balance of accuracy, low power consumption, and a practical form factor for everyday use.

What Problem Does This Paper Address?

The paper tackles the challenge of enabling rich, natural 3D hand interactions in mobile environments. Current approaches often fall short: data gloves are cumbersome and limit tactile feedback, environmental sensors require setup and are not mobile, and depth cameras are too power-hungry and bulky for widespread mobile adoption. Existing wearable systems often rely on simpler sensing (e.g., IR proximity, muscle activity) or 2D vision, limiting the fidelity of interaction. The authors aim to bridge this gap by providing a low-power, self-contained, wrist-worn system capable of recovering the full 3D pose of the user's hand, opening up possibilities for intuitive interactions on the go.

Methods

Digits employs a multi-modal sensing approach integrated into a compact wrist-worn device:

  • IR Laser Line Projection: A thin IR laser line is projected across the hand, intersecting with fingers. By triangulating these intersections with a fixed camera, a 3D point for each finger is recovered. This forms the basis of a forward kinematics (FK) model that reconstructs a basic 3D hand skeleton.
  • Diffuse IR LEDs: These LEDs illuminate the hand, allowing for the detection of finger separation and the estimation of surface normals and coarse depth for fingertips. This is achieved by modeling light falloff from the LEDs.
  • Combined Kinematic Models:
    • Simple FK Model: Leverages the laser line intersections and biomechanical constraints to reconstruct a 3D hand pose. This model is efficient but has limitations in capturing complex joint movements.
  • New Inverse Kinematics (IK) Model: This more advanced model fuses fingertip detection (from LEDs) with laser line intersection data. It uses an energy function that minimizes the difference between observed data and estimated hand poses, allowing for more independent articulation of finger joints (MCP, PIP, DIP) and a more accurate, higher-DOF hand reconstruction.
  • IMU: An Inertial Measurement Unit (IMU) is included to approximate wrist and forearm motions, though this was not explicitly used in the primary evaluation.

Datasets

Not discussed in the paper. The evaluation primarily used ground truth data from a Vicon motion tracking system for comparison and asked participants to mimic on-screen rendered hand postures.

Key Insights and Ideas

  • Wrist-worn is the sweet spot: The paper argues that a wrist-worn device offers a compelling balance of form factor, power efficiency, and interaction potential for mobile 3D hand tracking, avoiding the drawbacks of gloves or environmental sensors.
  • Fusion is key for fidelity: Combining different sensing modalities (laser line for 3D points, LEDs for surface details) and leveraging biomechanical constraints through both FK and IK models is crucial for achieving robust and high-fidelity 3D hand pose recovery from limited sensor data.
  • Kinematic models matter: The paper demonstrates how sophisticated kinematic models, particularly the novel IK approach, can significantly improve the accuracy and expressiveness of hand tracking by accounting for inter-joint dependencies and fusing sparse sensor inputs.
  • Beyond the screen: Digits enables interactions that extend beyond the immediate display of mobile devices, allowing for 3D spatial interactions in the air or for controlling larger displays, and even enabling eyes-free interactions.

Key Results

  • Accuracy: Preliminary evaluation showed mean errors in PIP joint angles to be relatively small, generally less than 9 degrees across all fingers and gestures. Index finger accuracy was particularly high (1.8° to 2.9°).
  • Repeatability: Intraclass correlation coefficients (ICC) ranged from 0.66 to 0.97, indicating moderate to strong correlation in joint-angle measurements across repetitions and participants, suggesting good system repeatability.
  • Acquisition Time: Participants improved their ability to match reference poses over time, with mean acquisition time decreasing significantly from the first block (4417 ms) to the second (3981 ms).
  • Power Consumption: The system draws less than 400mW, significantly lower than current generation Kinect cameras (3.4-5W).
  • Weight: The device weighs around 75g (124g with IMU), comparable to or lighter than standard watches.
  • Qualitative: The system demonstrated utility in various scenarios, including 3D spatial interaction with mobile devices, eyes-free interaction, and gaming. The IK model showed improved accuracy in recovering poses that were challenging for the FK model.

Limitations and Future Work

  • Occlusions: Vision-based tracking is susceptible to occlusions from crossed fingers, overly bent thumbs, or handheld objects. The authors suggest gesture design and more advanced finger separation techniques to mitigate this.
  • Wrist/Forearm Tracking: The current implementation does not explicitly model wrist bend and rotations. While the IMU is present, its integration for full pose reconstruction is future work.
  • Miniaturization: While the current prototype is wearable, it's still tethered to a PC. Future work involves miniaturizing the device into a standalone unit or integrating it into existing wearables.
  • Computational Cost: While designed for low power, the processing pipeline still requires computational resources. Further optimization or hardware acceleration could be explored.
  • Emergent Interactions: The authors suggest exploring interactions involving the non-instrumented hand, using the device as a trackpad, or leveraging it for more subtle, eyes-free interactions.

Summary updated

BibTeX
@inproceedings{kim2012digits,
  author = {David Kim and Otmar Hilliges and Shahram Izadi and Alex Butler and Jiawen Chen and Iason Oikonomidis and Patrick Olivier},
  title = {{Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor}},
  booktitle = {UIST},
  year = {2012},
  doi = {10.1145/2380116.2380139},
}
Figure