Gesture Typing on Virtual Tabletop: Effect of Input Dimensions on Performance

Antoine Loriette, Roderick Murray-Smith, Sebastian Stein, John WilliamsonISS 2017 · DOI

TL;DR

This paper explores gesture typing on virtual tabletops using depth cameras, enabling touch interaction on any flat surface. They developed a machine learning model for accurate fingertip detection and conducted a user study to understand how input surface size and aspect ratio affect typing performance, finding that larger surfaces can be used effectively with reduced input rates.

What Problem Does This Paper Address?

Traditional touchscreens are ubiquitous, but they have limitations, especially for users with physical impairments or in situations where touch interaction is difficult (e.g., dirty hands). Virtual surfaces, created using depth cameras, offer a flexible alternative, allowing for customizable input areas. However, accurately classifying touches on these surfaces and understanding how the physical characteristics of these virtual surfaces impact user performance, particularly for complex tasks like typing, remains an open challenge. This paper specifically investigates the trade-offs between input surface dimensions and gesture typing performance.

Methods

The core of the system relies on two main components:

  • Fingertip Touch Classification: The authors employ a supervised learning approach to classify fingertip touches. They trained a binary neural network classifier using features derived from the z-values of detected fingertip point clouds. This approach aims to be invariant to orientation and shape. The model was trained on a dataset of fingertip interactions captured over 6 minutes, totaling 10,800 frames, and validated using a 3-fold cross-validation strategy.
  • User Study Design: A user study was conducted to evaluate typing performance under different virtual surface configurations. Participants performed a gesture typing task on various virtual tabletop setups, comparing them to a direct interaction on a tablet. The study manipulated input size and aspect ratio, measuring error rate and input rate (Words Per Minute - WPM). They also collected qualitative data through the NASA Task Load Index (NASA-TLX) to assess perceived workload.

Datasets

  • Fingertip Touch Classification Dataset: A custom dataset was created by video-recording 6 minutes of interactions, resulting in 10,800 frames. This dataset was balanced for touching and hovering frames and included out-of-range frames. It was used to train and validate the fingertip touch classification model.
  • User Study Data: The user study involved 12 participants performing a typing task. For each participant, data was collected across 5 different experimental conditions (combinations of device, orientation, and size), with each trial involving typing 20 words. This resulted in a total of 1200 trials, with approximately 6,000 to 8,400 samples recorded.

Key Insights and Ideas

  • Virtual Surfaces as a Viable Alternative: The research demonstrates that gesture typing on virtual tabletops is a feasible interaction method, offering advantages over traditional touchscreens, particularly for users who might struggle with capacitive touch.
  • Machine Learning for Robust Touch Detection: Using a supervised learning approach for fingertip touch classification proved effective, achieving a good average AUC (0.96) for unseen users, suggesting the model generalizes well.
  • Size vs. Performance Trade-off: The study highlights that users can achieve similar error rates on virtual surfaces that are significantly larger than the physical tablet display, albeit at a reduced input rate. This suggests a potential for larger, more comfortable interaction areas.
  • Indirect Interaction as a Limiting Factor: While surface size and aspect ratio influence performance, the paper suggests that the indirect nature of the interaction (where the user's finger doesn't directly touch the display) might be the primary bottleneck for achieving higher input rates, rather than the physical dimensions of the virtual surface itself.
  • User Preference for Direct Interaction: Despite the potential of virtual surfaces, participants consistently ranked the direct interaction on a tablet as their preferred method, indicating that usability and learnability are still key considerations.

Key Results

  • Touch Classification Performance: The fingertip touch classification model achieved an average AUC of 0.96 across a 3-fold cross-validation, indicating good generalization.
  • Error Rate: Novice users achieved the same error rate on virtual surfaces at least twice the size of the visual feedback compared to a control tablet condition.
  • Input Rate: Participants typed at approximately half the input rate on larger virtual surfaces compared to the tablet. The input rate was largely independent of surface size across the tested range.
  • Optimal Size: An input size between A5 and A4 was found to offer the best trade-off between performance and user preference.
  • Qualitative Feedback: Participants found the virtual surface interaction (specifically OP4) to be more physically demanding and frustrating than the tablet interaction.

Limitations and Future Work

  • Novice Users: The study focused on novice users. Future work could explore how experienced typists perform with this system and whether the observed limitations persist.
  • Indirect Interaction Bottleneck: The paper posits that the indirect nature of the interaction is a limiting factor. Further research could investigate methods to mitigate this, such as improved visual feedback or exploring tactile feedback to aid pointer control.
  • Texture and Tactile Feedback: Some participants mentioned the texture of the surface impacting their interaction. The authors suggest exploring tactile feedback as a potential avenue for improving control and performance.
  • Model Improvement: While the current touch classification is effective, the authors note the potential for using more advanced models like convolutional neural networks for feature extraction.

Summary updated

BibTeX
@inproceedings{loriette2017gesture,
  author = {Antoine Loriette and Roderick Murray-Smith and Sebastian Stein and John Williamson},
  title = {{Gesture Typing on Virtual Tabletop: Effect of Input Dimensions on Performance}},
  booktitle = {ISS},
  year = {2017},
  doi = {10.1145/3132272.3135074},
}
Figure