TL;DR
Ego4o (Omni) is a novel framework that unifies 3D human motion capture and natural language motion understanding using sparse, everyday wearable sensors (like a smartwatch, phone, or smart glasses). By combining 1–3 random IMUs with an optional egocentric camera, it not only tracks full-body poses robustly under missing modalities but also establishes a unique feedback loop where AI-generated text descriptions are fed back into the system to significantly improve 3D pose accuracy.
What Problem Does This Paper Address?
Capturing and understanding human motion from a first-person (egocentric) perspective is highly valuable for AR/VR, sports, and healthcare. However, consumer wearables provide fragmented data: a smartwatch tracks only one wrist, a smartphone tracks only the hip, and smart glasses cameras suffer from frequent body self-occlusions.
Existing methods typically demand a fixed, single-modality setup (e.g., exactly six IMUs or a continuous video stream). They break down when a user takes off a device or turns off their camera due to privacy concerns. The authors of Ego4o tackle this real-world messiness by designing a framework capable of handling highly flexible, intermittent, and multi-modal inputs (images, sparse IMUs, and text) without sacrificing accuracy.
Methods
The Ego4o framework consists of two primary components operating over a shared discrete motion space:
- Part-Aware Motion Representation: Instead of encoding the entire human body into a single holistic codebook, the authors divide the body into six independent joint groups (head, root, left/right arms, left/right legs) using a part-aware Vector Quantized-Variational Autoencoder (VQ-VAE). This allows the network to perform "text-infilling" style predictions—inferring the movement of unmonitored limbs (like legs) based on data from monitored ones (like a wrist IMU).
- Multi-Modal Transformer Encoder: This module maps IMU sequences (acceleration and rotation), egocentric images (via CLIP features), and text descriptions into the part-aware motion codebook. To handle missing sensors at runtime, the model is trained with a random masking strategy across all modalities.
- Motion-Aware LLM Fine-Tuning: The discrete motion codes are treated as "motion tokens" and fed alongside egocentric images into a fine-tuned Large Language Model (LLaVA/Vicuna-7B). This allows the system to generate highly detailed natural language descriptions of the action.
- The Feedback Loop: Crucially, when human-provided text descriptions are missing, the text generated by the LLM can be fed back into the multi-modal encoder as a conditioning signal, resolving pose ambiguities caused by sparse sensor data.
Datasets
The model was trained and evaluated using three key datasets:
- Nymeria: A massive, recently released dataset containing multimodal egocentric daily motions in the wild, complete with 3D ground truth poses, egocentric video, IMU data, and fine-grained text descriptions.
- DIP-IMU: Used to benchmark IMU-based motion capture accuracy against existing state-of-the-art methods.
- AMASS: A large clean database of human motion used to pre-train the part-aware VQ-VAE codebook.
Key Insights and Ideas
- Cross-Modal Complementarity: The authors observe that different sensors excel at different things. Images provide environmental context (seeing a desk implies sitting), IMUs provide precise kinematic details (distinguishing a smash from a block in table tennis), and text provides high-level semantic constraints.
- Text as a Regularizer for Physics: One of the most striking "aha moments" is that generating text descriptions actually improves downstream 3D pose estimation. Natural language acts as a powerful prior that helps the model disambiguate noisy or static IMU signals.
- Part-Based Codebooks Prevent Error Propagation: By isolating body parts into separate codebooks, noisy data from a single swinging arm doesn't corrupt the positional tracking of the legs or head.
Key Results
Motion Capture Performance
On the Nymeria dataset, using a highly sparse setup (1–3 IMUs), Ego4o significantly outperformed the baseline IMUPoser:
- Ego4o (Full Multimodal): Achieved a Mean Per Joint Position Error (MPJPE) of 84.82 mm and a Procrustes-aligned error (PA-MPJPE) of 62.33 mm.
- IMUPoser: Achieved an MPJPE of 105.7 mm.
- Even when images and text were completely disabled (Ego4o-IMU), the architecture still outperformed the baseline, achieving 95.86 mm MPJPE.
Motion Understanding Performance
When generating natural language descriptions from motion tokens, Ego4o vastly outperformed specialized motion-to-text models like TM2T and MotionGPT, boosting the BERTScore from ~14.09 to 30.13 and Bleu@1 from ~42.22 to 53.83.
Efficiency
The framework is highly optimized for real-world deployment:
- Motion Capture: Runs at 8.2 to 8.6 ms per frame, easily clearing the bar for real-time tracking.
- Memory Footprint: The motion capture module consumes less than 1 GB of GPU memory.
Limitations and Future Work
While powerful, the authors note a few limitations:
- Sequence Latency: The system processes motion sequences rather than single frames, which introduces a slight inherent latency that could impact highly time-sensitive online applications.
- Conversational Depth: The fine-tuned LLM currently has a limited capacity for multi-round, interactive conversations about the user's motion.
Future work will focus on utilizing advanced instructional fine-tuning to enable richer, multi-turn dialogue and moving toward a true "human foundational model" capable of seamlessly interacting with users via text, audio, and physical movement.
Summary updated