HiSync: Spatio-Temporally Aligning Hand Motion from Wearable IMU and On-Robot Camera for Command Source Identification in Long-Range HRI

Chengwen Zhang, Chun Yu, Borong Zhuang, Haopeng Jin, Qingyang Wan, Zhuojun Li, Zhe He, Zhoutong Ye, Yu Mei, Chang Liu, Weinan Shi, Yuanchun Shi

TL;DR

HiSync is a novel optical-inertial fusion framework that solves the "who is commanding me?" problem in long-range human-robot interaction (up to 34 meters). By mapping hand motions from both a wearable IMU and an on-robot camera into the frequency domain, it accurately identifies the true command source even when the user's hand is reduced to a few blurred pixels.

What Problem Does This Paper Address?

In public spaces, directing a robot from a distance—such as summoning a robotaxi across a parking lot or calling a service drone—presents a fundamental challenge called Command Source Identification (CSI). When multiple people are gesturing simultaneously, how does a robot know who actually issued the command?

At close ranges (<10 meters), vision-only models can easily distinguish users by their appearance or fine-grained hand features. However, at long ranges (≥10 meters), visual data degrades severely. A user's hand might occupy a tiny 10×10 pixel region, rendering state-of-the-art pose estimators completely unreliable.

Figure 2: Visual Ambiguity at a Distance of 34 m. Figure (A) shows a real sample from our dataset (1920 × 1080 resolution). The inset highlights that the hand region occupies fewer than 10 × 10 pixels. Even with a manually zoomed-in view of the detection result like (B), YOLOv11x [34] fails to identify the hand. This demonstrates the inherent visual ambiguity in long-range interactions.

While pairing a wearable device (like a smartwatch or smart ring) provides clean inertial data, matching this data with the robot's camera stream is incredibly difficult due to wireless transmission delays, clock drift, and packet jitter. HiSync tackles these visual ambiguities and data noise challenges to enable robust, long-range CSI.

Methods

Instead of trying to match noisy, unsynchronized visual and inertial signals directly in the time domain, HiSync transforms the problem into the frequency domain.

Figure 1: Demonstration of HiSync. Multiple people perform similar gestures at a distance of 34 m. The figure illustrates the application scenario: Person 1 (blue link) controls the quadruped, while Person 2 (orange link) simultaneously controls the drone. Other bystanders act as visual distractors. Each robot receives an inertial stream from its paired device. HiSync enables the robot to identify its bound command issuer, effectively rejecting distractors regardless of their kinematic similarity.

The architecture consists of two main stages:

  1. Motion Feature Extractor:
  2. Visual Branch: Uses a two-stage pipeline (YOLOv11x + YOLOv11x-pose) to isolate a hand Region of Interest (ROI). It then computes dense optical flow using VideoFlow-MOP and mean-pools the flow within the ROI to estimate 2D image-plane hand velocity.
  3. Inertial Branch: Estimates 3D world-frame linear acceleration from a 150 Hz 9-DoF IMU using Madgwick AHRS orientation filtering and a low-pass Butterworth filter.
  4. Spectral Analysis: Converts both 2D velocities and 3D accelerations into Power Spectral Densities (PSD) alongside a compact set of descriptors (peak height, peak frequency, spectral clarity, entropy, frequency spacing, and SNR).
  5. CSINet (Command Source Identification Network):
  6. Quality-Aware Feature Modulation: Employs a Feature Linear Modulation (FiLM) operator to dynamically down-weight frequency channels corrupted by noise or occlusion.
  7. IMU-Anchored Cross-Modal Attention: Uses multi-head attention to softly align the visual flow sequences to the IMU time reference, neutralizing synchronization errors.
  8. Scale-Aware Multi-Window Fusion: Aggregates similarity scores across multiple Discrete Fourier Transform (DFT) window lengths using weights conditioned on the user's distance proxy.

Datasets

Because no public datasets existed for long-range multi-user CSI, the authors curated their own:

  • Formative Study Dataset: Elicited a natural gesture set from 12 participants across three robot embodiments (quadruped, wheeled cobot, drone).
  • HiSync Dataset: Features 38 participants, yielding 9,465 gesture sequences (452,055 total RGB frames). Data was captured from three camera perspectives (upward, eye-level, downward) across distances ranging from 3 to 34 meters. It includes synchronized tri-axial IMU data from three hand locations (wrist, palm, and index finger base).
Figure 5: Illustration of the User-defined Gesture Set.

Key Insights and Ideas

  • 2D Flow as a 3D Proxy: The authors discovered that 2D tangential motion from a monocular camera correlates exceptionally well (Pearson correlation coefficient > 0.925) with 3D radial motion magnitude. This means a standard 2D camera is mathematically sufficient to capture the essential motion characteristics of 3D gestures.
  • The Power of the Frequency Domain: Shifting from the time domain to the frequency domain provides inherent invariance to global time shifts. A constant delay between the IMU and camera simply manifests as a linear phase shift in the frequency domain, which can be easily handled, making the system highly tolerant to wireless network jitter.
  • Distance-Adaptive Windowing: The optimal temporal window size changes with distance. Larger windows excel at far ranges by capturing global motion characteristics (like overall swing frequency), whereas shorter windows are better at close ranges for capturing fine-grained, personalized force dynamics.

Key Results

  • State-of-the-Art Performance: Across the entire 3–34m range, HiSync achieved an overall CSI accuracy of 97.82%.
  • Extreme Distance Resilience: In three-person scenes at the maximum distance of 34 meters, HiSync maintained a 92.32% CSI accuracy, outperforming the previous state-of-the-art time-domain fusion method (VIPL) by a massive 48.44% margin.
  • Real-Robot Deployment: When deployed live on a Unitree Go2 quadruped robot in public spaces with adversarial "mimic" distractors trying to trick the system, HiSync achieved 90.9% accuracy at 30 meters with an average response time of 3.88 seconds.

Limitations and Future Work

While highly effective, the authors note several limitations ripe for future research:

  • Crowd Density: The evaluation environments had a relatively low crowd density (averaging ~3 people per frame). Performance in highly congested environments like packed airports or shopping malls remains to be tested.
  • Robot Ego-Motion: The current pipeline is sensitive to the robot's own movement. The system benefits significantly when the robot pauses before performing identification. Integrating visual-inertial SLAM or odometry to compensate for ego-motion is a critical next step.
  • Multi-IMU Scaling: The current setup assumes only one target user transmits IMU data to a specific robot. Scenarios where multiple users simultaneously send IMU streams to a shared communal robot have not yet been experimentally validated.

Summary updated

Abstract

Long-range Human-Robot Interaction (HRI) remains underexplored. Within it, Command Source Identification (CSI) - determining who issued a command - is especially challenging due to multi-user and distance-induced sensor ambiguity. We introduce HiSync, an optical-inertial fusion framework that treats hand motion as binding cues by aligning robot-mounted camera optical flow with hand-worn IMU signals. We first elicit a user-defined (N=12) gesture set and collect a multimodal command gesture dataset (N=38) in long-range multi-user HRI scenarios. Next, HiSync extracts frequency-domain hand motion features from both camera and IMU data, and a learned CSINet denoises IMU readings, temporally aligns modalities, and performs distance-aware multi-window fusion to compute cross-modal similarity of subtle, natural gestures, enabling robust CSI. In three-person scenes up to 34m, HiSync achieves 92.32% CSI accuracy, outperforming the prior SOTA by 48.44%. HiSync is also validated on real-robot deployment. By making CSI reliable and natural, HiSync provides a practical primitive and design guidance for public-space HRI. https://github.com/OctopusWen/HiSync
BibTeX
@inproceedings{zhang2026hisync,
  author = {Chengwen Zhang and Chun Yu and Borong Zhuang and Haopeng Jin and Qingyang Wan and Zhuojun Li and Zhe He and Zhoutong Ye and Yu Mei and Chang Liu and Weinan Shi and Yuanchun Shi},
  title = {{HiSync: Spatio-Temporally Aligning Hand Motion from Wearable IMU and On-Robot Camera for Command Source Identification in Long-Range HRI}},
  booktitle = {CHI},
  year = {2026},
  doi = {10.1145/3772318.3790345},
}
Figure