TL;DR
HiSync is a novel optical-inertial fusion framework that solves the "who is commanding me?" problem in long-range human-robot interaction (up to 34 meters). By mapping hand motions from both a wearable IMU and an on-robot camera into the frequency domain, it accurately identifies the true command source even when the user's hand is reduced to a few blurred pixels.
What Problem Does This Paper Address?
In public spaces, directing a robot from a distance—such as summoning a robotaxi across a parking lot or calling a service drone—presents a fundamental challenge called Command Source Identification (CSI). When multiple people are gesturing simultaneously, how does a robot know who actually issued the command?
At close ranges (<10 meters), vision-only models can easily distinguish users by their appearance or fine-grained hand features. However, at long ranges (≥10 meters), visual data degrades severely. A user's hand might occupy a tiny 10×10 pixel region, rendering state-of-the-art pose estimators completely unreliable.
While pairing a wearable device (like a smartwatch or smart ring) provides clean inertial data, matching this data with the robot's camera stream is incredibly difficult due to wireless transmission delays, clock drift, and packet jitter. HiSync tackles these visual ambiguities and data noise challenges to enable robust, long-range CSI.
Methods
Instead of trying to match noisy, unsynchronized visual and inertial signals directly in the time domain, HiSync transforms the problem into the frequency domain.
The architecture consists of two main stages:
- Motion Feature Extractor:
- Visual Branch: Uses a two-stage pipeline (YOLOv11x + YOLOv11x-pose) to isolate a hand Region of Interest (ROI). It then computes dense optical flow using VideoFlow-MOP and mean-pools the flow within the ROI to estimate 2D image-plane hand velocity.
- Inertial Branch: Estimates 3D world-frame linear acceleration from a 150 Hz 9-DoF IMU using Madgwick AHRS orientation filtering and a low-pass Butterworth filter.
- Spectral Analysis: Converts both 2D velocities and 3D accelerations into Power Spectral Densities (PSD) alongside a compact set of descriptors (peak height, peak frequency, spectral clarity, entropy, frequency spacing, and SNR).
- CSINet (Command Source Identification Network):
- Quality-Aware Feature Modulation: Employs a Feature Linear Modulation (FiLM) operator to dynamically down-weight frequency channels corrupted by noise or occlusion.
- IMU-Anchored Cross-Modal Attention: Uses multi-head attention to softly align the visual flow sequences to the IMU time reference, neutralizing synchronization errors.
- Scale-Aware Multi-Window Fusion: Aggregates similarity scores across multiple Discrete Fourier Transform (DFT) window lengths using weights conditioned on the user's distance proxy.
Datasets
Because no public datasets existed for long-range multi-user CSI, the authors curated their own:
- Formative Study Dataset: Elicited a natural gesture set from 12 participants across three robot embodiments (quadruped, wheeled cobot, drone).
- HiSync Dataset: Features 38 participants, yielding 9,465 gesture sequences (452,055 total RGB frames). Data was captured from three camera perspectives (upward, eye-level, downward) across distances ranging from 3 to 34 meters. It includes synchronized tri-axial IMU data from three hand locations (wrist, palm, and index finger base).
Key Insights and Ideas
- 2D Flow as a 3D Proxy: The authors discovered that 2D tangential motion from a monocular camera correlates exceptionally well (Pearson correlation coefficient > 0.925) with 3D radial motion magnitude. This means a standard 2D camera is mathematically sufficient to capture the essential motion characteristics of 3D gestures.
- The Power of the Frequency Domain: Shifting from the time domain to the frequency domain provides inherent invariance to global time shifts. A constant delay between the IMU and camera simply manifests as a linear phase shift in the frequency domain, which can be easily handled, making the system highly tolerant to wireless network jitter.
- Distance-Adaptive Windowing: The optimal temporal window size changes with distance. Larger windows excel at far ranges by capturing global motion characteristics (like overall swing frequency), whereas shorter windows are better at close ranges for capturing fine-grained, personalized force dynamics.
Key Results
- State-of-the-Art Performance: Across the entire 3–34m range, HiSync achieved an overall CSI accuracy of 97.82%.
- Extreme Distance Resilience: In three-person scenes at the maximum distance of 34 meters, HiSync maintained a 92.32% CSI accuracy, outperforming the previous state-of-the-art time-domain fusion method (VIPL) by a massive 48.44% margin.
- Real-Robot Deployment: When deployed live on a Unitree Go2 quadruped robot in public spaces with adversarial "mimic" distractors trying to trick the system, HiSync achieved 90.9% accuracy at 30 meters with an average response time of 3.88 seconds.
Limitations and Future Work
While highly effective, the authors note several limitations ripe for future research:
- Crowd Density: The evaluation environments had a relatively low crowd density (averaging ~3 people per frame). Performance in highly congested environments like packed airports or shopping malls remains to be tested.
- Robot Ego-Motion: The current pipeline is sensitive to the robot's own movement. The system benefits significantly when the robot pauses before performing identification. Integrating visual-inertial SLAM or odometry to compensate for ego-motion is a critical next step.
- Multi-IMU Scaling: The current setup assumes only one target user transmits IMU data to a specific robot. Scenarios where multiple users simultaneously send IMU streams to a shared communal robot have not yet been experimentally validated.
Summary updated