Introduction
Artificial intelligence models often struggle with "catastrophic forgetting"—the tendency to abruptly overwrite old knowledge when learning new tasks. While this has been extensively studied in static image classification, it remains a significant challenge for generative temporal sequences, such as human motion. Generating motion requires a model to internalize the complex dynamics of body movement over time, not just recognize a static label.
The authors address this through a framework called CL2Gen (Continual Learning-to-Generate). The goal is to enable a model to learn a sequence of distinct movements (e.g., walking, then jumping, then running) while retaining the ability to generate high-quality, recognizable sequences of all previously learned actions.
The BI-CTVAE Architecture
To solve the CL2Gen problem, the researchers developed the BI-CTVAE (Biologically-Inspired Conditional Temporal Variational Autoencoder). This model mimics the brain's memory consolidation processes, specifically the way the hippocampus and motor cortex "replay" memories during sleep.
The architecture is built on several key pillars:
- Recurrent Backbone: Utilizing Gated Recurrent Units (GRUs) to process the temporal nature of motion, which is more biologically plausible than non-causal Transformer architectures.
- Latent Mixture-of-Gaussians (GMM): Instead of a single distribution, the model assigns a specific "mode" (trainable mean and variance) to each action class. This structural prior reduces "crosstalk" between different motions in the latent space.
- Generative Replay: Before learning a new task, the model generates synthetic examples of previously learned actions. It then trains on a mixture of these "memories" and new real data, stabilizing internal representations.
- Multi-faceted Loss Function: The training incorporates Reconstruction Loss, Vertex Loss (using a pretrained SMPL 3D body model for physical accuracy), and an Auxiliary Loss from a frozen classifier that "critiques" the generated motions.
Experimental Results
The model was evaluated using the HumanAct12 dataset, which contains 1,191 motion sequences across 12 categories. The categories were split into six sequential tasks (two classes per task).
The results demonstrate the effectiveness of the biologically-inspired approach:
- Accuracy Retention: The BI-CTVAE achieved a final classification accuracy of 78.6% on generated samples after learning all 12 classes. In contrast, a baseline model without replay suffered total catastrophic forgetting, collapsing to just 14.8% accuracy.
- Closing the Gap: The performance of the BI-CTVAE is remarkably close to the "Gold Standard" offline model (trained on all data at once), which achieved 84.0%.
- Quality and Diversity: Metrics like Fréchet Inception Distance (FID) confirmed that the generated sequences remained high-quality and diverse throughout the continual learning process.
Conclusion and Future Work
The BI-CTVAE demonstrates that generative replay is a powerful tool for maintaining stability in temporal models without the need to store original training data. By leveraging a GMM latent space and biologically-inspired replay, the model successfully navigates the trade-off between plasticity (learning new things) and stability (remembering old things). Future research will explore scaling this approach to larger datasets and applying it to other domains like robotics and video synthesis.
Reference: Ott, J., & Liu, S.-C. (2023). Biologically-Inspired Continual Learning of Human Motion Sequences. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP).
Summary updated