TL;DR
This paper presents a gesture recognition system for assisted living that leverages commodity smartwatches and smartphones. By using accelerometer data from a smartwatch, the system can recognize simple wrist gestures, offering a more natural and less intrusive way for individuals to interact with their environment. The study evaluates three recognition methods, finding SAX to be the most efficient and accurate.
What Problem Does This Paper Address?
Gesture recognition is a powerful tool for human-computer interaction, offering intuitive ways to control devices and convey information. However, many existing systems rely on complex setups like video cameras or specialized wearable sensors (e.g., gloves with sensors). These approaches can be intrusive, expensive, or limit user mobility. The authors aim to address this by exploring the use of readily available and affordable commodity smartwatches, which are equipped with accelerometers and wireless communication, to enable gesture recognition without requiring specialized hardware or restricting user movement. This is particularly relevant for assisted living applications, where a simple, natural, and non-intrusive interaction method can significantly improve the quality of life for elderly individuals or those with physical limitations.
Methods
The core of the proposed system involves capturing accelerometer data from a smartwatch and processing it on a connected smartphone. The authors explored three distinct methodologies for gesture recognition:
- Fast Fourier Transformation (FFT) Method: This approach applies a low-pass filter to the raw accelerometer data, resamples it to a fixed length, and then computes the FFT. The first 21 coefficients from each axis are retained as features. Gesture similarity is then calculated using the mean Euclidean distance between these feature vectors.
- Geometric Method: This method transforms the time series by replacing each sample with the sum of its value and all preceding values. The data is then scaled to fit within a normalized cube. Similar to the FFT method, gesture similarity is determined by the mean Euclidean distance between the transformed time series.
- SAX (Symbolic Aggregate approXimation) Method: This technique transforms time series into symbolic strings. The accelerometer data is first normalized, then undergoes Piecewise Aggregate Approximation (PAA) to reduce its length, and finally, it's discretized into a string of symbols using a chosen alphabet. The distance between two gestures is calculated based on the distance between their corresponding symbols.
The system architecture involves a Pebble smartwatch for data capture and an Android smartphone for processing, user interface, and database management. Communication between the devices is handled via Bluetooth. A key design choice for the smartwatch application is the use of a "tap event" to initiate and terminate gesture recording, allowing for spontaneous and natural gesture input.
Datasets
The paper does not introduce a new benchmark dataset. Instead, it utilizes a custom dataset collected by asking four individuals to perform a set of six 3D natural gestures (illustrated in Figure 1). Each gesture was repeated multiple times by each user. The dataset was collected in two phases, with a few days between them, to assess both user-dependent and user-independent recognition capabilities.
Key Insights and Ideas
- Commodity Smartwatches as Powerful Sensors: The study demonstrates that readily available smartwatches, with their embedded accelerometers, are capable of capturing sufficient data for reliable gesture recognition, eliminating the need for specialized and potentially costly hardware.
- SAX's Efficiency and Accuracy: The SAX method emerged as the most effective, achieving the highest accuracy in both user-dependent and user-independent scenarios. Its ability to represent time series as symbolic strings seems particularly well-suited for handling the inherent variability in sensor data.
- User-Independent Recognition is Achievable: The SAX method's strong performance in user-independent recognition (where the system is trained on one user and tested on others) is a significant finding, suggesting its potential for broader applicability without extensive individual training.
- Tap Event for Natural Interaction: Using a simple tap event to start and stop gesture recording provides a natural and intuitive way for users to interact with the system, enhancing usability, especially for the target demographic.
- Battery Life Considerations: While smartwatches offer convenience, battery consumption is a crucial factor. The paper provides an estimation of battery life, suggesting that even with frequent gesture use, the autonomy remains acceptable for daily use.
Key Results
- User-Dependent Recognition:
- Geometric method: 93% accuracy.
- FFT method: 95% accuracy.
- SAX method: 99% accuracy.
- User-Independent Recognition:
- Geometric method: 84% average accuracy.
- FFT method: 71% average accuracy.
- SAX method: 96% average accuracy.
- Usability Assessment: A modified System Usability Scale (SUS) survey with 20 elderly users (age 63-72) yielded a score of 78/100, indicating a positive reception and perceived usability of the system.
- Battery Consumption: The system's use resulted in a 10% battery drop after approximately 780 gestures, with an estimated autonomy of over 6 days even with 80 gestures per day.
Limitations and Future Work
- Gesture Set Simplicity: The gestures used for evaluation were simple and primarily involved wrist movements. The authors acknowledge that the system's suitability depends on the distinctiveness of gestures, and further research is needed for more complex gesture sets.
- Gravity Contribution: While accelerometer data includes gravity, the current set of gestures did not fully exploit this for diversification. Future work could explore gestures that leverage gravity more effectively.
- Dynamic Update of Library: The current system requires explicit training. The authors suggest exploring dynamic updates to the gesture library to adapt to changes in user gesture patterns over time.
- Universal Vocabulary: The development of a universal vocabulary for gesture commands is proposed as a future direction, enabling a standardized way to control various devices.
- Gender-Related Differences: The usability assessment did not reveal gender-related differences, but the authors intend to investigate this further in future evaluations.
- Education and Technology Exposure: While these factors are known to influence technology acceptance, they were not explicitly included in the questionnaire for this study but are planned for future, more thorough evaluations.
Summary updated