TL;DR
This paper introduces two novel systems for smartphones: one that automatically recognizes which hand a user is operating the device with, and another that detects when a user switches hands. By doing so, these systems pave the way for dynamic user interface adjustments, significantly improving usability on large-screen smartphones without requiring manual input.
What Problem Does This Paper Address?
As smartphone screens have grown larger, single-handed operation has become increasingly difficult, especially for users with smaller hands. This leads to a frustrating user experience where essential buttons or controls might be out of reach. While some solutions exist, they often require manual user input to select the operating hand or confirm hand-switching. This paper addresses the challenge of creating an automatic and seamless system that can infer the user's operating hand and detect hand-changes, enabling proactive UI adjustments to enhance usability.
Methods
The researchers propose two distinct systems:
- Operating Hand Recognition (from Touchscreen Traces): This system analyzes the patterns of user interactions on the touchscreen. It extracts various features from touchscreen traces, including:
- Length Features: Total length of the trace.
- Velocity Features: Maximum, average velocity, and standard deviation.
- Displacement Features: Total and maximum X and Y displacements.
-
Shape Features: Root Mean Squared Error (RMSE) for smoothness, curvature, and curve convex orientation. These features are fed into a Random Forest (RF) classifier to determine if the user is operating with their left hand, right hand, or another (e.g., both hands). The system also infers operation modes (e.g., holding and operating, holding only) based on the sequence of hand classifications.
-
Hand-Changing Process Detection (from Accelerometer and Gyroscope Data): This system leverages the smartphone's inertial sensors. It first segments the sensor data using an end-points detection algorithm. Then, it extracts features across three categories:
- Time Domain Features: Basic shape characteristics like Max, Mean, Min, and Range.
- Statistics Features: Kurtosis (peakedness/flatness) and distribution characteristics.
- Frequency Domain Features: Centroid, DC amplitude, Decrease, Flux, Peak Amplitude, Roll-off, and Spread, derived from FFT analysis. These features are then used to train a classifier (also explored with RF and other algorithms like MLP) to distinguish between hand-changing events and other daily activities (walking, running, etc.). The paper also explores Dynamic Time Warping (DTW) as a feature extraction method for this system, though it's ultimately deemed too computationally expensive.
Datasets
The paper utilizes datasets collected from participants using smartphones.
- Operating Hand Recognition: Data was collected from 14 participants generating traces in four directions (right, left, up, down) using their right/left thumbs and index fingers. The paper mentions experiments with varying numbers of training samples (from 5 to 40) and trees in the RF classifier.
- Hand-Changing Process Detection: The experimental dataset includes:
- 500 instances of "Left Hand to Right Hand" changes.
- 500 instances of "Right Hand to Left Hand" changes.
- A large set of segments (130,050) representing various daily activities (walking, running, going upstairs/downstairs) collected over seven days. For DTW evaluation, a smaller dataset was used: 20 instances for each hand-changing direction in the template library and 500 instances for each direction in the test set, plus 50,000 daily activity segments over four days.
Key Insights and Ideas
- Touchscreen Traces as a Rich Source of Information: The paper demonstrates that subtle characteristics of how a user draws on a touchscreen (speed, smoothness, curvature) are highly indicative of which hand is being used. This is a novel approach to inferring hand dominance.
- Inertial Sensors for Dynamic Hand-Switching: Accelerometer and gyroscope data can effectively capture the distinct motion patterns associated with physically passing a smartphone between hands, differentiating it from other common activities.
- Automatic UI Adjustment is Feasible: The core idea is that by automatically knowing the operating hand and detecting changes, smartphones can proactively adjust their interfaces (e.g., repositioning buttons) to optimize for single-handed use, leading to a more intuitive and less error-prone experience.
- Random Forest as a Strong Performer: The RF classifier consistently showed excellent performance in both systems, balancing accuracy and computational efficiency.
- DTW's Computational Cost: While DTW can be effective for sequence matching, its high computational cost makes it impractical for real-time applications on smartphones, highlighting the importance of efficient feature engineering.
Key Results
- Operating Hand Recognition: Achieved 94.1% precision and 94.1% True Positive Rate (TPR) with a 2.6% False Positive Rate (FPR) using a single touchscreen trace.
- Hand-Changing Process Detection: Achieved 93.9% precision and 93.7% TPR with a 0.7% FPR using accelerometer and gyroscope data.
- Classifier Performance: Random Forest (RF) and Multi-Layer Perceptron (MLP) were top performers for hand-changing detection, with RF being chosen for its better efficiency.
- Feature Extraction Efficiency: Most features for operating hand recognition could be computed within 17 ms. For hand-changing detection, feature extraction was more time-consuming (around 4.53 ms per segment), but still feasible.
- Dimension Reduction: Combining RF with Linear Discriminant Analysis (LDA) for hand-changing detection reduced features to 2 dimensions while maintaining high accuracy (93.9% precision, 93.7% TPR).
Limitations and Future Work
- UI Adjustment Specificity: The paper acknowledges that while they provide the recognition systems, the actual adjustment strategies for user interfaces are application-specific and require more sophisticated design and artistic input. They propose their solution could be integrated into operating systems for developers.
- Energy Consumption: While the systems are designed to be efficient, the paper notes that touchscreen activation and sensor usage do consume energy, though they argue it's comparable to or less than existing smartphone usage patterns.
- DTW's Impracticality: The high time consumption of DTW for hand-changing detection led to its exclusion from the final system, suggesting a need for more efficient sequence matching algorithms or feature representations.
- Future Work: The authors suggest that their systems can work independently or jointly to further improve accuracy and reduce false UI adjustments. They also imply that integrating this technology into operating systems would be the next logical step for widespread adoption.
Summary updated