Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input

Jian Wang, Rishabh Dabral, Diogo Luvizon, Zhe Cao, Lingjie Liu, Thabo Beeler, Christian Theobalt

TL;DR

Ego4o (Omni) is a novel framework that unifies 3D human motion capture and natural language motion understanding using sparse, everyday wearable sensors (like a smartwatch, phone, or smart glasses). By combining 1–3 random IMUs with an optional egocentric camera, it not only tracks full-body poses robustly under missing modalities but also establishes a unique feedback loop where AI-generated text descriptions are fed back into the system to significantly improve 3D pose accuracy.

What Problem Does This Paper Address?

Capturing and understanding human motion from a first-person (egocentric) perspective is highly valuable for AR/VR, sports, and healthcare. However, consumer wearables provide fragmented data: a smartwatch tracks only one wrist, a smartphone tracks only the hip, and smart glasses cameras suffer from frequent body self-occlusions.

Existing methods typically demand a fixed, single-modality setup (e.g., exactly six IMUs or a continuous video stream). They break down when a user takes off a device or turns off their camera due to privacy concerns. The authors of Ego4o tackle this real-world messiness by designing a framework capable of handling highly flexible, intermittent, and multi-modal inputs (images, sparse IMUs, and text) without sacrificing accuracy.

Figure 1. Our method can use an egocentric image and 1-3 IMU sensors from wearable devices to accurately predict human motion and generate motion descriptions. Motion descriptions, when available, can also enhance motion capture accuracy. Ego4o supports flexible input combinations, functioning with or without images, or with varied IMU placements.

Methods

The Ego4o framework consists of two primary components operating over a shared discrete motion space:

  1. Part-Aware Motion Representation: Instead of encoding the entire human body into a single holistic codebook, the authors divide the body into six independent joint groups (head, root, left/right arms, left/right legs) using a part-aware Vector Quantized-Variational Autoencoder (VQ-VAE). This allows the network to perform "text-infilling" style predictions—inferring the movement of unmonitored limbs (like legs) based on data from monitored ones (like a wrist IMU).
  2. Multi-Modal Transformer Encoder: This module maps IMU sequences (acceleration and rotation), egocentric images (via CLIP features), and text descriptions into the part-aware motion codebook. To handle missing sensors at runtime, the model is trained with a random masking strategy across all modalities.
  3. Motion-Aware LLM Fine-Tuning: The discrete motion codes are treated as "motion tokens" and fed alongside egocentric images into a fine-tuned Large Language Model (LLaVA/Vicuna-7B). This allows the system to generate highly detailed natural language descriptions of the action.
  4. The Feedback Loop: Crucially, when human-provided text descriptions are missing, the text generated by the LLM can be fed back into the multi-modal encoder as a conditioning signal, resolving pose ambiguities caused by sparse sensor data.
Overview of the Ego4o framework architecture

Datasets

The model was trained and evaluated using three key datasets:

  • Nymeria: A massive, recently released dataset containing multimodal egocentric daily motions in the wild, complete with 3D ground truth poses, egocentric video, IMU data, and fine-grained text descriptions.
  • DIP-IMU: Used to benchmark IMU-based motion capture accuracy against existing state-of-the-art methods.
  • AMASS: A large clean database of human motion used to pre-train the part-aware VQ-VAE codebook.

Key Insights and Ideas

  • Cross-Modal Complementarity: The authors observe that different sensors excel at different things. Images provide environmental context (seeing a desk implies sitting), IMUs provide precise kinematic details (distinguishing a smash from a block in table tennis), and text provides high-level semantic constraints.
  • Text as a Regularizer for Physics: One of the most striking "aha moments" is that generating text descriptions actually improves downstream 3D pose estimation. Natural language acts as a powerful prior that helps the model disambiguate noisy or static IMU signals.
  • Part-Based Codebooks Prevent Error Propagation: By isolating body parts into separate codebooks, noisy data from a single swinging arm doesn't corrupt the positional tracking of the legs or head.

Key Results

Motion Capture Performance

On the Nymeria dataset, using a highly sparse setup (1–3 IMUs), Ego4o significantly outperformed the baseline IMUPoser:

  • Ego4o (Full Multimodal): Achieved a Mean Per Joint Position Error (MPJPE) of 84.82 mm and a Procrustes-aligned error (PA-MPJPE) of 62.33 mm.
  • IMUPoser: Achieved an MPJPE of 105.7 mm.
  • Even when images and text were completely disabled (Ego4o-IMU), the architecture still outperformed the baseline, achieving 95.86 mm MPJPE.

Motion Understanding Performance

When generating natural language descriptions from motion tokens, Ego4o vastly outperformed specialized motion-to-text models like TM2T and MotionGPT, boosting the BERTScore from ~14.09 to 30.13 and Bleu@1 from ~42.22 to 53.83.

Efficiency

The framework is highly optimized for real-world deployment:

  • Motion Capture: Runs at 8.2 to 8.6 ms per frame, easily clearing the bar for real-time tracking.
  • Memory Footprint: The motion capture module consumes less than 1 GB of GPU memory.

Limitations and Future Work

While powerful, the authors note a few limitations:

  • Sequence Latency: The system processes motion sequences rather than single frames, which introduces a slight inherent latency that could impact highly time-sensitive online applications.
  • Conversational Depth: The fine-tuned LLM currently has a limited capacity for multi-round, interactive conversations about the user's motion.

Future work will focus on utilizing advanced instructional fine-tuning to enable richer, multi-turn dialogue and moving toward a true "human foundational model" capable of seamlessly interacting with users via text, audio, and physical movement.

Summary updated

Abstract

This work focuses on tracking and understanding human motion using consumer wearable devices, such as VR/AR headsets, smart glasses, cellphones, and smartwatches. These devices provide diverse, multi-modal sensor inputs, including egocentric images, and 1-3 sparse IMU sensors in varied combinations. Motion descriptions can also accompany these signals. The diverse input modalities and their intermittent availability pose challenges for consistent motion capture and understanding. In this work, we present Ego4o (o for omni), a new framework for simultaneous human motion capture and understanding from multi-modal egocentric inputs. This method maintains performance with partial inputs while achieving better results when multiple modalities are combined. First, the IMU sensor inputs, the optional egocentric image, and text description of human motion are encoded into the latent space of a motion VQ-VAE. Next, the latent vectors are sent to the VQ-VAE decoder and optimized to track human motion. When motion descriptions are unavailable, the latent vectors can be input into a multi-modal LLM to generate human motion descriptions, which can further enhance motion capture accuracy. Quantitative and qualitative evaluations demonstrate the effectiveness of our method in predicting accurate human motion and high-quality motion descriptions.
BibTeX
@inproceedings{wang2025ego,
  author = {Jian Wang and Rishabh Dabral and Diogo Luvizon and Zhe Cao and Lingjie Liu and Thabo Beeler and Christian Theobalt},
  title = {{Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Input}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02111},
}
Figure