EventHOI: Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses

Le Zhang, Kailai Cui, Hao Chen, Vlad Roznyatovskiy, Jianzhong Zhang, Anhong Guo, Kang G. Shin, Ke SunarXiv 2026 · arXiv

TL;DR

EventHOI is a resource-efficient system for smartglasses that uses ultra-low-power event sensing and IMUs to selectively trigger high-power RGB cameras only when hand-object interactions (HOIs) occur. By combining intelligent egomotion cancellation with saliency-guided compression, it reduces always-on power consumption by up to 10.4× and bandwidth by 2.9×, making all-day activity logging feasible over standard Bluetooth Low Energy (BLE).

What Problem Does This Paper Address?

Smartglasses have the potential to be powerful "memory assistants," logging our daily interactions—like taking medication or locking the door—to help us later. However, two major bottlenecks prevent all-day operation:

  1. Energy/Thermal Limits: Continuous RGB recording consumes massive power (200–600 mW), causing devices to overheat and draining batteries in as little as 1.5 hours.
  2. Bandwidth Constraints: Streaming high-quality video typically requires Wi-Fi, which is too energy-intensive for wearables. Lower-power radios like BLE have very limited throughput (often <0.3 Mbps), which is usually insufficient for standard video streams.

Existing solutions often rely on periodic sampling or other sensors (like microphones or thermal cameras) that are either power-hungry themselves or easily confused by noisy environments.

Methods

The authors propose EventHOI, which introduces a low-power "visual sentinel" to gate the operation of the high-power RGB camera.

Figure 1: EventHOI uses low-power event sensing to (1) detect and selectively trigger the high-power RGB camera and (2) compress RGB uplink for HOI understanding.

The system architecture consists of three main stages:

  • Egomotion Cancellation: Event cameras are sensitive to motion, but on a head-mounted device, most "motion" comes from the wearer's head turns or walking. EventHOI uses a 6-axis IMU to estimate this egomotion. It employs a lightweight LSTM to detect walking and a gyroscope-based reprojection to cancel out head rotations, leaving only "residual" motion likely caused by hands and objects.
  • HOI Motion Detection: The system analyzes the residual motion to identify clusters that look like hand-object interactions. It uses a hysteresis-based strategy to avoid frequent on-off toggling of the RGB camera, ensuring temporal continuity for downstream models.
  • HOI-Aware Video Encoding: Once an interaction is detected, the system doesn't just send the whole frame. It uses the event data to identify a Region of Interest (RoI) and crops the RGB frame accordingly. This "saliency-guided" compression allows the stream to fit within the narrow bandwidth of BLE.

Datasets

The researchers evaluated EventHOI using two primary sources:

  • Nymeria Dataset: A large-scale egocentric benchmark. They used a 50-hour subset featuring 97 subjects across diverse daily scenarios. Since Nymeria lacks event camera data, they used the v2e simulator to generate event frames from RGB video.
  • Real-World Testbed: The authors built a custom smartglasses prototype equipped with a real event camera (Prophesee GENX320), an RGB camera, and an IMU. They collected a new 20-hour dataset from 10 participants performing natural daily routines in homes and offices.

Key Insights and Ideas

  • Event Frames as Gating Signals: Event cameras are >60× more energy-efficient than RGB sensors because they only report pixel-level intensity changes. This makes them ideal "always-on" triggers.
  • Stability Prior: A key insight for egomotion cancellation is that meaningful HOIs rarely happen during large translational movements (like walking). Most interactions occur when the wearer's body is relatively stable, allowing the system to safely disable recording during vigorous movement.
  • VLM Robustness: The authors found that modern Vision-Language Models (VLMs) are surprisingly robust to low frame rates (1 FPS) and spatial cropping, provided the essential hand-object cues are preserved. This allows for aggressive temporal and spatial downsampling.
Figure 3: Preliminary study on VLM-based HOI understand- ing for frame rate and uplink bandwidth.

Key Results

  • Power Savings: EventHOI reduces always-on backend power consumption by 5.8× to 10.4× compared to continuous RGB and YOLO-based baselines.
  • Bandwidth Efficiency: It reduces uplink bandwidth by 2.9×, bringing the stream down to less than 130 kbps. This makes stable video streaming over BLE a reality.
  • Accuracy Preservation: Despite the massive data reduction, the system incurs only a negligible loss (~1.2%) in HOI recognition accuracy when evaluated with state-of-the-art VLMs like Qwen3-VL and Gemini.
  • Latency: The end-to-end on-device pipeline (from event detection to RGB capture) operates within ~500 ms, well within the requirements for 1 FPS logging.

Limitations and Future Work

The authors identify two main failure modes for their current motion-based detector:

  • Background Distractors: Dynamic content on TVs or smartphones and movements from bystanders can cause "false positive" triggers.
  • Repetitive Textures: Highly textured backgrounds (like keyboards or tiled floors) can cause aliasing in the optical flow algorithm, leading to incorrect motion estimates.

Future work involves integrating lightweight appearance or semantic cues (perhaps from low-resolution monochrome frames) to better distinguish true hand-object interactions from background noise.

Summary updated

Abstract

Hand-object interactions (HOIs) can be captured by smartglasses to learn how people manipulate objects in daily life. Automatic logging of these interactions from an egocentric perspective provides time-stamped evidence of daily actions for applications like memory assistance. However, continuous RGB capture and wireless transmission exceed the energy and bandwidth budgets of smartglasses. We present EventHOI, an event-driven sensing and compression system that couples selective RGB capture with HOI-aware encoding to enable egocentric logging over Bluetooth Low Energy (BLE) while preserving downstream HOI logging performance. Our key insight is to use low-power event sensing both to determine when RGB capture is needed and to identify where visual detail must be preserved. A lightweight IMU-based egomotion cancellation pipeline suppresses egomotion-induced events so that residual HOI motion can guide RGB triggering and spatial compression. We build a smartglasses prototype and conduct an end-to-end evaluation of EventHOI on a 50-hour egocentric benchmark with 97 subjects and a 20-hour real-world dataset we collect from 10 participants, together covering a broad range of daily-life scenarios and environments. Compared to RGB-only baselines, EventHOI reduces always-on backend power consumption by 5.8-10.4x and uplink bandwidth by 2.9x, with minimal HOI recognition accuracy loss. The resulting RGB uplink of less than 130 kbps fits within commodity BLE bandwidth budgets, making stable BLE video streaming feasible for egocentric HOI logging. Demo video and project website: https://eventhoi.github.io
BibTeX
@misc{zhang2026eventhoi,
  author = {Le Zhang and Kailai Cui and Hao Chen and Vlad Roznyatovskiy and Jianzhong Zhang and Anhong Guo and Kang G. Shin and Ke Sun},
  title = {{EventHOI: Event-Driven Resource-Efficient Hand-Object Interaction Logging on Smartglasses}},
  year = {2026},
  eprint = {2611.09089},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.09089},
}
Figure