Unistroke gestures on multi-touch interaction

Yingying Jiang, Feng Tian, Xiaolong Zhang, Wei Liu, Guozhong Dai, Hongan Wang2012 · DOI

TL;DR

This paper tackles the challenge of recognizing flexible multi-touch gestures on large surfaces like tabletops, where users naturally employ multiple fingers and casual touchdowns. They introduce a "key stroke extraction" method that simplifies complex multi-finger inputs into a single representative stroke, significantly improving gesture recognition accuracy compared to traditional methods.

What Problem Does This Paper Address?

Modern multi-touch devices, especially large tabletops, offer a more natural and flexible interaction style. Unlike the precise, single-finger gestures common on smartphones, users on tabletops often use multiple fingers simultaneously, with casual touch-downs and liftoffs. This flexibility, while enhancing user experience, poses a significant challenge for gesture recognition systems, which are often designed for simpler, single-stroke inputs. Existing algorithms struggle to accurately interpret these complex, multi-finger movements, leading to recognition errors. This paper aims to bridge this gap by developing a method to robustly recognize these flexible multi-touch unistroke gestures.

Methods

The core of the paper's contribution lies in its "key stroke extraction" method. Instead of directly processing all raw touch points from a multi-finger gesture, the authors first extract a representative "key stroke." This key stroke is essentially a simplified representation of the gesture, derived from the multiple touch points. They explore three techniques for this extraction:

  • Removing temporally overlapped points: This method keeps only the earliest touch point when multiple fingers touch down simultaneously.
  • Taking the centroid of temporally overlapped points: This approach calculates the average position of all simultaneously touching fingers to form a single point.
  • Averaging adjacent points: This technique smooths out the gesture by averaging consecutive points, reducing the impact of minor variations from multiple fingers.

Once the key stroke is extracted, a standard unistroke gesture recognizer (specifically, the $1 recognizer) is applied. This approach effectively treats a complex multi-finger gesture as a single, simplified stroke, making it more amenable to traditional recognition algorithms.

Datasets

The paper describes two user studies:

  1. Initial User Study: Conducted on a VR VISION tabletop, this study collected data on five basic gestures (line, circle, triangle, rectangle, multiline) drawn by 12 graduate students. Participants were encouraged to use any method they preferred, with no visual feedback provided to encourage natural drawing styles. This study helped establish the synchronized nature of multi-finger movements in gestures.
  2. Evaluation Study: This study used a dataset of 16 unistroke gestures, similar to those used in prior research [10]. 12 graduate students performed each gesture ten times, resulting in a total of 1920 gestures. This dataset was used to compare the proposed key stroke extraction methods against a baseline method without extraction.

Key Insights and Ideas

  • Synchronized Multi-Finger Movement: Despite using multiple fingers, the movements are often synchronized and produce similar stroke shapes, suggesting that a single representative stroke can capture the essence of the gesture.
  • Key Stroke Extraction is Crucial: The paper strongly advocates for pre-processing multi-touch inputs by extracting a key stroke before applying a gesture recognizer. This simplification is key to handling the inherent flexibility and noise in multi-finger interactions.
  • Comparable Performance of Extraction Methods: The three proposed key stroke extraction techniques (temporal overlap removal, centroid, averaging) all performed comparably well and significantly outperformed the baseline method. This suggests flexibility in choosing an extraction method based on specific needs.
  • Tabletop Interaction is Inherently More Flexible: The authors acknowledge that their reported accuracies are lower than some prior studies, attributing this to the more flexible nature of tabletop gestures and the lack of visual feedback during data collection, which leads to less "perfect" gestures.

Key Results

  • Improved Accuracy: The key stroke extraction methods (T2, T3, T4) consistently achieved significantly lower error rates compared to the baseline method (T1) that did not use key stroke extraction. For instance, with just one training sample, error rates for T2, T3, and T4 were around 6%, while T1 was 17.81%.
  • No Significant Difference Between Extraction Methods: While all key stroke extraction methods outperformed the baseline, there was no statistically significant difference in performance among T2, T3, and T4 themselves.
  • Time Efficiency: The time efficiency of the proposed methods was comparable to the baseline, with average recognition times around 5.8-6ms per gesture when using one template.

Limitations and Future Work

  • Lower Absolute Accuracy: The authors acknowledge that their absolute recognition accuracies are lower than some previous studies, attributing this to the inherent flexibility of tabletop gestures and the lack of visual feedback during data collection.
  • Extending to Multi-Stroke Gestures: The current work focuses on unistroke gestures. Future work will extend the algorithm to handle multi-stroke gestures.
  • Multi-User and Small Screen Devices: The authors also plan to explore the application of their work to multi-user interactions and gestures on small screen devices.
  • Disambiguation of Ambiguous Gestures: The paper briefly touches upon the challenge of distinguishing between accidental and deliberate multi-touch gestures, suggesting the integration of multi-touch specific algorithms or learning-based disambiguation methods.

Summary updated

Abstract

Gesture inputs on multi-touch tabletops usually involve multiple fingers (more than two) and casual touchdowns or liftoffs of fingers. This flexibility of touch gestures allows more natural user interaction, but also poses new challenges for accurate recognition of multi-touch gestures. To address these challenges, we propose a new approach to recognize flexible multi-touch stroke gestures on tabletops. Based on a user study on multi-touch unistroke gestures, we develop a gesture recognition method by extracting key strokes embedded in flexible multi-touch input. Our evaluation study result shows that this method can greatly improve the recognition accuracy of flexible multi-touch unistroke gestures on tabletops.
BibTeX
@misc{jiang2012unistroke,
  author = {Yingying Jiang and Feng Tian and Xiaolong Zhang and Wei Liu and Guozhong Dai and Hongan Wang},
  title = {{Unistroke gestures on multi-touch interaction}},
  year = {2012},
  doi = {10.1145/2166966.2166982},
}
Figure