MoML: Online Meta Adaptation for 3D Human Motion Prediction

Xiaoning Sun, Huaijiang Sun, Bin Li, Dong Wei, Weiqing Li, Jianfeng Lu

Introduction

Predicting human motion in short-term horizons is a fundamental challenge for applications ranging from autonomous driving to human-robot interaction. Traditional state-of-the-art models typically operate in a "static" paradigm: they are trained offline on fixed datasets and their parameters remain frozen during deployment. This approach struggles with "concept drift"—the real-world phenomenon where human behavior changes dynamically, such as a person suddenly transitioning from walking to sitting.

Comparison between existing frozen methods and the proposed online adaptive setting.

MoML (Online Meta Adaptation) addresses this by treating motion prediction as a continuous learning task. Instead of relying on a static model, MoML uses recent prediction errors as feedback to update lightweight "adapter" modules in real-time, allowing the system to adjust to new motion contexts as they occur.

Methodology

The MoML framework is built upon three core technical pillars designed to enable efficient, real-time adaptation:

  1. Bilevel Optimization: Inspired by Model-Agnostic Meta-Learning (MAML), the training process involves an "outer loop" that learns a robust generic initialization and an "inner loop" that performs rapid online updates to specialize the model to the current motion stream.
  2. MoAdapter Blocks: To maintain computational efficiency and prevent catastrophic forgetting, the authors introduce lightweight modules called MoAdapters. These include FC-MoAdapter (fully connected) and GC-MoAdapter (graph convolutional), the latter of which incorporates skeletal structure priors. During online inference, only these adapter parameters are updated.
  3. Fast-MoML: For high-frequency applications, a variant called Fast-MoML utilizes a closed-form ridge regression solution for the final layer, enabling nearly instantaneous adaptation without the need for iterative gradient descent.

Experimental Evaluation

The framework was evaluated across three major benchmarks: Human3.6M, CMU-Mocap, and 3DPW. These datasets cover a wide range of actions, from professional acting to real-world "in the wild" scenarios.

Key Results

MoML consistently improved the performance of various baseline architectures, including LTD, SPGSN, and MotionMixer. On the Human3.6M dataset, MoML reduced the Mean Per Joint Position Error (MPJPE) across nearly all categories. For instance, in the "discussion" category at a 400ms horizon, the SPGSN baseline improved from 67.1mm to 63.9mm.

Qualitative comparison of motion prediction. Red boxes indicate errors in baseline models (LTD, SPGSN), while green boxes show improvements after applying MoML.

Qualitative analysis reveals that MoML effectively corrects errors during motion transitions where frozen models typically fail. By treating prediction errors as informative signals rather than just failures, the model gains the "common sense" to adjust its internal state for subsequent frames.

Discussion and Conclusion

The shift from static windows to a streaming data paradigm represents a significant step toward practical deployment in robotics. MoML demonstrates that selective plasticity—separating generic kinematic parameters from adaptive context-specific ones—provides a superior balance of stability and flexibility.

While adaptation offers clear benefits in dynamic scenarios, its gains are less pronounced in highly repetitive or static motions. Future research may focus on intelligent triggering mechanisms to decide exactly when to adapt, further optimizing the balance between computational overhead and predictive accuracy.

Summary updated

Abstract

In the academic field, the research on human motion pre-diction tasks mainly focuses on exploiting the observed in-formation to forecast human movements accurately in the near future horizon. However, a significant gap appears when it comes to the application field, as current models are all trained offline, with fixed parameters that are inher-ently suboptimal to handle the complex yet ever-changing nature of human behaviors. To bridge this gap, in this pa-per, we introduce the task of online meta adaptation for hu-man motion prediction, based on the insight that finding “smart weights” capable of swift adjustments to suit dif-ferent motion contexts along the time is a key to improving predictive accuracy. We propose MoML, which ingeniously borrows the bilevel optimization spirit of model-agnostic meta-learning, to transform previous predictive mistakes into strong inductive biases to guide online adaptation. This is achieved by our MoAdapter blocks that can learn er-ror information by facilitating efficient adaptation via a few gradient steps, which fine-tunes our meta-learned “smart” initialization produced by the generic predictor. Considering real-time requirements in practice, we further propose Fast-MoML, a more efficient variant of MoML that features a closed-form solution instead of conventional gradient up-date. Experimental results show that our approach can ef-fectively bring many existing offline motion prediction mod-els online, and improves their predictive accuracy.
BibTeX
@inproceedings{sun2024moml,
  author = {Xiaoning Sun and Huaijiang Sun and Bin Li and Dong Wei and Weiqing Li and Jianfeng Lu},
  title = {{MoML: Online Meta Adaptation for 3D Human Motion Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00105},
}
Figure