REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning

Jihyun Lee, Weipeng Xu, Alexander Richard, Shih-En Wei, Shunsuke Saito, Shaojie Bai, Te-Li Wang, Minhyuk Sung, Tae-Kyun Kim, Jason M. Saragih

Abstract

We present REWIND (Real-Time Egocentric Whole-Body Motion Diffusion), a one-step diffusion model for real-time, high-fidelity human motion estimation from egocentric image inputs. While existing methods for egocentric whole-body motion estimation are often non-real-time and acausal due to iterative refinement, REWIND operates in a fully causal and real-time manner. To enable real-time inference, we introduce cascaded body-hand denoising diffusion, which models the correlation between body and hand motions in a fast, feed-forward manner, and diffusion distillation, which enables high-quality estimation with a single denoising step. Our model uses a modified Transformer architecture designed to causally model output motions while enhancing generalizability. Additionally, REWIND supports identity-conditioned motion estimation using a small set of pose exemplars, further enhancing quality. Extensive experiments show that REWIND significantly outperforms existing baselines.

Introduction: The Challenge of Egocentric Tracking

In Augmented and Virtual Reality (AR/VR), tracking a user's full body and hands from their own perspective—known as egocentric motion estimation—is a significant technical hurdle. Headset-mounted cameras typically have a limited field of view, leading to truncated limbs and frequent self-occlusions. Previous state-of-the-art solutions, such as EgoWholeMocap, relied on acausal diffusion models that required future frames for refinement and multiple iterative steps for denoising. This high latency (often over 2 seconds per frame) makes them unsuitable for real-time applications like gaming or telepresence.

Egocentric input views showing the limited field of view and self-occlusion challenges.

The REWIND Framework

REWIND addresses these limitations through three core innovations designed to bridge the gap between high-fidelity generative modeling and real-time performance:

  1. Cascaded Body-Hand Denoising: The model first estimates the body pose and then uses the upper-body position as conditioning to estimate hand poses. This hierarchical approach leverages the natural correlation between arm and hand movements to resolve ambiguities when hands are partially occluded.
  2. Causal Relative-Temporal Transformer: By employing Rotary Positional Encoding (RoPE) and windowed attention, REWIND models motion causally (relying only on past frames). This architecture allows for tracking over indefinite motion lengths without the performance degradation seen in models using absolute time encodings.
  3. Diffusion Distillation: To achieve real-time speeds, the authors utilized Score Distillation Sampling (SDS). This process distills the knowledge of a multi-step diffusion model into a "student" model capable of producing high-quality poses in a single feed-forward pass.
  4. Exemplar-Based Identity Conditioning: The framework can optionally incorporate a small set of 3D pose exemplars from a specific user. This identity prior helps the model adapt to individual body proportions and movement styles, significantly boosting tracking accuracy.

Experimental Evaluation

Datasets

The model was trained and evaluated on ColossusEgo, a massive real-world dataset containing over 2.8 million frames from 500 subjects, and UnrealEgo, a synthetic dataset used to test generalization across environments.

Performance and Accuracy

REWIND achieves a breakthrough in processing speed, running at approximately 32ms per frame (over 30 FPS) on a single GPU. This is an 80x speedup over previous diffusion-based methods like EgoWholeMocap (2576ms).

Qualitative results of REWIND showing estimated poses compared to ground truth across various subjects and motions.

In terms of accuracy, REWIND reduced the Mean Per Joint Position Error (MPJPE) on the ColossusEgo dataset from ~62mm to ~54mm for the body and from ~26mm to ~21mm for hands. When identity conditioning was applied, body error further decreased to 48.45mm, demonstrating the value of personalized priors.

Conclusion

REWIND represents a major step forward for egocentric motion tracking, proving that the generative power of diffusion models can be harnessed for real-time applications through distillation and hierarchical conditioning. While challenges like self-penetration and extreme occlusion remain, the framework sets a new standard for high-fidelity, causal whole-body estimation in AR/VR environments.

Summary updated

Abstract

We present REWIND (Real-Time Egocentric Whole-Body Motion Diffusion), a one-step diffusion model for real-time, high-fidelity human motion estimation from egocentric image inputs. While an existing method for egocentric whole-body (i.e., body and hands) motion estimation is non-real-time and acausal due to diffusion-based iterative motion refinement to capture correlations between body and hand poses, REWIND operates in a fully causal and real-time manner. To enable real-time inference, we introduce (1) cascaded body-hand denoising diffusion, which effectively models the correlation between egocentric body and hand motions in a fast, feed-forward manner, and (2) diffusion distillation, which enables high-quality motion estimation with a single denoising step. Our denoising diffusion model is based on a modified Transformer architecture, designed to causally model output motions while enhancing generalizability to unseen motion lengths. Additionally, REWIND optionally supports identity-conditioned motion estimation when identity prior is available. To this end, we propose a novel identity conditioning method based on a small set of pose exemplars of the target identity, which further enhances motion estimation quality. Through extensive experiments, we demonstrate that REWIND significantly outperforms the existing baselines both with and without exemplar-based identity conditioning.
BibTeX
@inproceedings{lee2025rewind,
  author = {Jihyun Lee and Weipeng Xu and Alexander Richard and Shih-En Wei and Shunsuke Saito and Shaojie Bai and Te-Li Wang and Minhyuk Sung and Tae-Kyun Kim and Jason M. Saragih},
  title = {{REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=3YFHxEsTTt},
}
Figure