Abstract
Gesture interaction is a natural way of communicating with a robot as an alternative to speech. Gesture recognition methods leverage optical flow in order to understand human motion. However, while accurate optical flow estimation (i.e., traditional) methods are costly in terms of runtime, fast estimation (i.e., deep learning) methods' accuracy can be improved. In this paper, we present a pipeline for gesture-based human-robot interaction that uses a novel optical flow estimation method in order to achieve an improved speed-accuracy trade-off. Our optical flow estimation method introduces four improvements to previous deep learning-based methods: strong feature extractors, attention to contours, midway features, and a combination of these three. This results in a better understanding of motion, and a finer representation of silhouettes. In order to evaluate our pipeline, we generated our own dataset, MIBURI, which contains gestures to command a house service robot. In our experiments, we show how our method improves not only optical flow estimation, but also gesture recognition, offering a speed-accuracy trade-off more realistic for practical robot applications.
Introduction
For robots to function effectively as domestic assistants, they must be capable of natural communication with human users. While speech recognition is a standard interface, it frequently encounters limitations in noisy environments or when interacting with users who have speech disabilities. Gesture-based interaction provides a vital alternative, yet the computational requirements for a robot to "see" and interpret motion in real-time are significant.
The standard approach for motion interpretation involves optical flow—the calculation of pixel movement between consecutive video frames. Historically, researchers have faced a difficult trade-off: traditional variational methods offer high accuracy but are too slow for real-time robotics, while early deep learning models (such as FlowNet) are fast but produce "blurry" motion maps. This lack of precision often results in the loss of fine silhouette details, leading to errors in gesture classification. This paper introduces a pipeline designed to bridge this gap, delivering high-fidelity motion estimation at speeds suitable for interactive robotics.
Proposed Methodology
The proposed pipeline processes RGB video through a novel optical flow estimator, which then feeds the resulting motion data into a Temporal Segment Network (TSN) for final gesture classification. The core innovation lies in four specific architectural enhancements to the optical flow estimation process:
- Advanced Feature Extractors: Rather than relying on basic convolutional layers, the architecture adapts sophisticated classification backbones—specifically ResNet, Inception, and ResNeXt—to extract more discriminative features from input frames.
- Attention Mechanism: A sigmoid-based attention module was introduced to force the network to prioritize the contours of moving objects, such as a user's hand or arm. This effectively sharpens the silhouette against the background, preserving critical shape information.
- Midway Module: To address the loss of accuracy that often occurs during the upscaling of motion "pyramids," the midway module focuses on mid-level scales. This ensures that motion patches remain uniform and prevents erroneous jumps in pixel velocity within a single moving object.
- Integrated Model (AttMidFlowNet): The flagship model combines these elements to produce high-fidelity motion maps that outperform standard deep learning baselines.
Experimental Setup and Datasets
The system was evaluated using four distinct datasets to test both raw flow accuracy and gesture recognition performance:
- MIBURI: A custom dataset developed for human-robot interaction (HRI), featuring 600 videos of 10 household commands (e.g., "follow-me," "bring-drink") recorded from a robot's perspective.
- ChaLearn IGR: A large-scale public benchmark for isolated gesture recognition containing over 47,000 videos.
- FlyingChairs & MPI Sintel: Standard synthetic datasets used to benchmark raw optical flow estimation metrics.
Key Results and Discussion
The experimental results demonstrate a significant improvement in both motion estimation and practical application:
- Optical Flow Accuracy: On the FlyingChairs benchmark, the proposed methods improved upon the FlowNetS baseline by 20% to 45%.
- Gesture Recognition Performance: On the MIBURI dataset, the TSN/AttMidFlowNeXt64 model achieved 98.4% accuracy, a dramatic increase from the 73.9% achieved by the standard baseline. On the more complex ChaLearn IGR dataset, the attention-based models improved recognition by 7% to 10%.
- Efficiency and Real-world Viability: A critical finding was the improvement in runtime. By replacing traditional flow methods (like TVL1) with optimized deep learning models, the system reduced total recognition time from approximately 27 seconds to under 10 seconds per gesture, all while maintaining or increasing accuracy.
A key insight from this research is that for gesture recognition, a sharp silhouette is often more critical than a low End Point Error (EPE). By prioritizing edge definition through attention mechanisms, the system achieves superior recognition even when the underlying flow mathematics are slightly less "perfect" than traditional, slower methods.
Conclusion
This work presents a high-performance gesture recognition pipeline that optimizes the speed-accuracy trade-off for HRI. By incorporating state-of-the-art feature extractors and attention mechanisms, the system provides the clarity needed for complex motion interpretation without the prohibitive computational costs of traditional methods. Future work will focus on integrating this pipeline into the Robot Operating System (ROS) for live testing and conducting user studies to evaluate its effectiveness compared to voice-controlled systems in domestic environments.
Summary updated