Abstract
A key step towards understanding human behavior is the prediction of 3D human motion. Successful solutions have many applications in human tracking, HCI, and graphics. Most previous work focuses on predicting a time series of future 3D joint locations given a sequence of 3D joints from the past. This Euclidean formulation generally works better than predicting pose in terms of joint rotations. Body joint locations, however, do not fully constrain 3D human pose, leaving degrees of freedom (like rotation about a limb) undefined. We instead predict a sparse set of locations on the body surface that correspond to motion capture markers. Given such markers, we fit a parametric body model to recover the 3D body of the person. These sparse surface markers also carry detailed information about human movement that is not present in the joints, increasing the naturalness of the predicted motions. Using the AMASS dataset, we train MOJO (More than Our JOints), which is a novel variational autoencoder with a latent DCT space that generates motions from latent frequencies. MOJO preserves the full temporal resolution of the input motion, and sampling from the latent frequencies explicitly introduces high-frequency components into the generated motion. We note that motion prediction methods accumulate errors over time, resulting in joints or markers that diverge from true human bodies. To address this, we fit the SMPL-X body model to the predictions at each time step, projecting the solution back onto the space of valid bodies. Quantitative and qualitative experiments show that our approach produces state-of-the-art results and realistic 3D body animations.
Introduction
Most human motion prediction research treats the body as a skeleton—a collection of 3D joint locations. While computationally efficient, this approach has two major flaws. First, joints do not fully constrain the body; for example, a skeleton doesn't tell you if a person's arm is rotating around the bone's axis. Second, as recursive models predict further into the future, errors accumulate, leading to "exploding skeletons" where limbs stretch or compress into non-human proportions.
To create truly realistic animations for VR, AR, or HCI, we need a representation that captures surface nuances and stays grounded in the reality of human anatomy. The authors argue that we are "more than our joints" and propose a system that shifts the focus to sparse surface markers.
The MOJO Framework
The researchers introduce MOJO, which utilizes three key innovations to improve the realism and stability of predicted motion:
1. Marker-Based Representation
Instead of joints, MOJO predicts the 3D locations of sparse markers on the body surface, similar to those used in professional motion capture. This provides richer constraints on body pose and shape, naturally coupling global translation and local pose. This makes the model less prone to "foot skating"—the immersion-breaking effect where a character's feet slide across the floor because global movement doesn't match the leg's stride.
2. Latent DCT Space
Standard generative models often compress an entire motion sequence into a single latent vector. MOJO instead uses a Discrete Cosine Transform (DCT) to decompose motion into different frequency bands. By sampling from these bands, the model can explicitly introduce high-frequency details (the natural nuance of real movement) while maintaining the low-frequency "intent" of the motion.
3. Recursive Projection
To prevent predicted markers from drifting into impossible configurations, MOJO uses a "projection" step during inference. At each time step, the model fits an expressive 3D body model (SMPL-X) to the predicted markers. This effectively "snaps" the noisy prediction back onto the manifold of valid human bodies before the next step is predicted, ensuring the output is always anatomically plausible.
Experiments and Results
The authors primarily utilize AMASS, a massive collection of motion capture datasets. They also evaluate on Human3.6M and HumanEva-I to compare against traditional skeleton-based state-of-the-art methods.
- State-of-the-Art Accuracy: MOJO outperforms existing stochastic models (like DLow) on standard benchmarks in both diversity and prediction accuracy.
- Stability: The recursive projection scheme virtually eliminated bone length deformation. While baseline models saw limbs stretch by over 70mm over time, MOJO kept deformations to roughly 1mm.
- Realism: In a perceptual study, users consistently rated MOJO's animations as more natural than joint-based baselines. MOJO also achieved a significantly lower foot skating ratio, particularly on complex datasets like BMLhandball.
Limitations and Future Work
The primary drawback of MOJO is its computational cost. The recursive projection step—fitting the SMPL-X model at every frame—is computationally expensive, running at roughly 2.27 seconds per frame. This makes the current implementation unsuitable for real-time applications like live gaming. Future work may involve "distilling" this projection step into a faster neural network to enable real-time performance while maintaining the high level of realism achieved here.
Summary updated