Biologically-Inspired Continual Learning of Human Motion Sequences

Joachim Ott, Shih-Chii Liu

TL;DR

Researchers have developed a biologically-inspired model called BI-CTVAE that allows AI to learn new human motion sequences (like jumping or walking) sequentially without forgetting previous ones. By mimicking the way the brain "replays" memories during sleep, the model maintains high generative accuracy, achieving 78% classification performance—nearly matching models trained on all data at once.

What Problem Does This Paper Address?

Most AI models suffer from catastrophic forgetting: when you teach them a new task, they abruptly overwrite the knowledge of previous tasks. While "Continual Learning" (CL) has been studied extensively for image classification, it is much less explored for generative temporal sequences, such as human motion.

Generating motion is harder than classifying it because the model must understand the underlying dynamics of how a body moves over time. The authors introduce a specific challenge called CL2Gen (Continual Learning-to-Generate), where the goal isn't just to remember a label, but to retain the ability to produce high-quality, recognizable sequences of an action even after learning many other distinct movements.

Methods

The authors propose the BI-CTVAE (Biologically-Inspired Conditional Temporal Variational Autoencoder). Its architecture and training strategy are built on several key pillars:

  • Recurrent Backbone: Unlike many modern models that use Transformers (which are non-causal and arguably less biologically plausible), this model uses GRUs (Gated Recurrent Units) to handle the temporal nature of motion.
  • Latent Mixture-of-Gaussians (GMM): Instead of forcing all actions into a single distribution in the latent space, the model assigns a specific "mode" (a trainable mean and variance) to each action class. This helps separate different motions in the model's "memory."
  • Generative Replay (The "Sleep" Mechanism): To prevent forgetting, the model uses generative replay. Before learning a new task, the model uses its current knowledge to generate "synthetic" examples of previously learned actions. It then trains on a mix of these synthetic "memories" and the new real data. This mimics how the hippocampus and motor cortex are thought to consolidate memories during sleep.
  • Multi-faceted Loss Function: The model is optimized using a combination of:
  • Reconstruction Loss: Ensuring the output looks like the input.
  • Vertex Loss: Using a pretrained SMPL (a 3D human body model) to ensure the physical mesh of the body is accurate.
  • Auxiliary Loss: A frozen, pretrained classifier "critiques" the generated motions, providing feedback on whether a generated "jump" actually looks like a jump.

Datasets

The study uses HumanAct12, a standard dataset for human motion. It contains 1,191 motion sequences across 12 categories (e.g., 'warm-up', 'walk', 'run', 'jump').

To test continual learning, the authors adapted this dataset by splitting the 12 categories into 6 sequential tasks (2 classes per task). The model must learn Task 1, then Task 2, and so on, without ever seeing the original data from Task 1 again.

Key Insights and Ideas

  • Generative Replay is Sufficient: The paper demonstrates that you don't need to store old data (which can be a privacy or storage concern). Generating "fake" versions of old tasks is enough to keep the model's internal representations stable.
  • The Power of the GMM Latent Space: By giving each class its own trainable parameters in the latent space, the model reduces "crosstalk" between actions. This structural prior makes it much easier for the model to distinguish between similar motions like "walking" and "running" during the replay phase.
  • Utility over Perfection: Interestingly, the generated replay samples don't have to be perfect to be useful. Even slightly "blurry" or low-quality generated motions provide enough signal to prevent the weights of the neural network from shifting too far away from the old tasks.
  • Bio-Plausibility Works: Using causal RNNs (GRUs) and a replay mechanism inspired by neuroscience resulted in a model that is not only more "brain-like" but also highly effective at a complex engineering task.

Key Results

  • Accuracy: The BI-CTVAE achieved a final classification accuracy of 78.6% on generated samples after learning all 12 classes.
  • Comparison: This is a massive improvement over a baseline model with no replay, which collapsed to 14.8% accuracy (pure catastrophic forgetting).
  • Closing the Gap: The performance is only 5.4% lower than a "Gold Standard" offline model that was trained on all data simultaneously.
  • FID Scores: The Fréchet Inception Distance (FID), which measures the quality and diversity of generated sequences, was significantly better in the BI-CTVAE compared to standard GRU-based continual learning approaches.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Replay Ratios: The performance is sensitive to the "replay class ratio" (how many synthetic samples are generated per new real sample). Higher ratios perform better but increase training time.
  • Dataset Diversity: The model was only tested on HumanAct12. Future work is needed to see if this approach scales to much larger, more complex datasets like NTU RGB+D.
  • Scenario Expansion: The authors suggest investigating the BI-CTVAE in other CL2Gen scenarios beyond human motion, potentially in robotics or video synthesis.

Reference: Ott, J., & Liu, S.-C. (2023). Biologically-Inspired Continual Learning of Human Motion Sequences. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP).

Summary updated

BibTeX
@inproceedings{ott2023biologically,
  author = {Joachim Ott and Shih-Chii Liu},
  title = {{Biologically-Inspired Continual Learning of Human Motion Sequences}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095490},
}
Figure