Introduction
In the quest for more natural human-robot interaction (HRI), researchers are moving away from traditional interfaces like joysticks and keyboards toward more intuitive modalities. One of the most fundamental human behaviors is pointing. While pointing seems simple, detecting it reliably in real-time is surprisingly difficult. Traditional methods often rely on hand-crafted rules—such as checking if an arm remains still at a certain angle—which are brittle and struggle with the noise of everyday movement (walking, climbing, moving objects).
This research explores turning simple pointing movements into a reliable digital trigger using wearable sensors and deep learning. The goal is to detect the onset of the gesture—the exact moment the user starts to point—with minimal delay, enabling a robot to respond immediately to a user's intent.
Methodology
The system utilizes two Inertial Measurement Units (IMUs) placed on the user's forearm and upper arm, streaming acceleration vectors and orientation quaternions at 50 Hz. Instead of manual thresholds, a 1D Convolutional Neural Network (CNN) is employed to learn the temporal dynamics of the movement.
The network processes 2.5-second windows of data (125 samples) to perform binary classification, distinguishing between the start of a pointing gesture and background activity. By focusing on temporal patterns rather than static poses, the system becomes more robust against false positives where a user might briefly hold their arm in a similar position during other tasks.
Dataset and Data Augmentation
Because no standard benchmark existed for gesture onset detection, a custom dataset was collected. It includes 12 recording sessions with 4 actors, totaling 85 minutes of data and over 460 pointing gestures. These gestures were performed amidst "distractor" activities like walking, climbing ladders, and moving objects to ensure real-world applicability.
A key innovation is the use of Data Augmentation (DA) to achieve rotational invariance. Since a pointing gesture looks similar regardless of the user's heading, the recorded orientation values were mathematically rotated around the vertical axis. This effectively multiplied the training data and taught the model to recognize gestures from any direction without requiring thousands of additional physical recordings.
Results
The system's performance was evaluated using the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC). The results demonstrate that the proposed data augmentation strategy significantly boosts performance.
- Baseline (No Augmentation): AUC of 0.889.
- With Data Augmentation: AUC rose to 0.921.
In real-time validation, the system successfully triggered true positives within 1 to 3 seconds of a prompt while maintaining a low false-positive rate during non-pointing activities.
Conclusion
This work demonstrates a successful proof-of-concept for human-drone interaction using wearable IMUs and 1D CNNs. Future work could explore more complex gesture varieties beyond binary detection and investigate the trade-offs between window length and detection latency to further refine the system's responsiveness.
Summary updated