Introduction
In the field of online learning, models must adapt to a continuous stream of data. A significant challenge in this paradigm is catastrophic forgetting, where new information overwrites previously learned knowledge. The standard solution is Experience Replay (ER), which maintains a buffer of past observations to periodically retrain the model. However, ER's memory requirements scale linearly with the amount of data, making it unsuitable for resource-constrained environments like IoT devices or embedded sensors.
This paper introduces Online Contrastive Divergence with Generative Replay (OCDGR). Instead of storing raw data, OCDGR uses a generative model to "dream up" synthetic past experiences. This approach achieves state-of-the-art performance with a constant memory footprint, effectively decoupling memory usage from the total volume of processed data.
The OCDGR Framework
The core innovation of OCDGR is the replacement of a physical data buffer with a generative process. The authors utilize a Restricted Boltzmann Machine (RBM) as the underlying architecture.
While standard RBMs are typically trained offline using Contrastive Divergence (CD), OCDGR adapts this for online streams through a two-phase cycle:
- Generative Replay: Upon receiving a new batch of data, the current RBM uses its internal weights to generate synthetic data points that represent its current knowledge of the past distribution.
- Model Update: The RBM is then updated using a combination of the new real data and these synthetic "replayed" samples.
This mechanism transforms the learning process into a Markov chain where the model's next state depends only on its current state and the incoming data, rather than an ever-growing external database.
Experimental Evaluation
The authors evaluated OCDGR against various ER baselines across nine real-world datasets, including MNIST and the UCI Evaluation Suite (covering domains like biology, internet data, and document classification). To test for maximum vulnerability to forgetting, they employed a "worst-case" scenario where data arrived in a sorted order (e.g., all '0's followed by all '1's in MNIST).
Key Results
- Performance and Stability: OCDGR outperformed traditional ER in 64.28% of tested scenarios. Notably, OCDGR's learning curves were more stable than ER, which often exhibited "jitter" as the buffer distribution shifted.
- Memory Efficiency: OCDGR maintains a constant memory footprint regardless of the number of data points processed. In the sorted MNIST task, OCDGR significantly outperformed memory-limited ER.
- Regularization Effect: In several instances, OCDGR outperformed ER even when ER was allowed "infinite" memory. This suggests that the generative process may act as a regularizer, providing a cleaner representation of the underlying data distribution than raw, noisy samples.
Conclusion and Future Work
OCDGR provides a robust framework for online learning without the overhead of data storage. By mimicking the biological process of memory reconstruction, it enables high-performance learning on low-power devices. Future research directions include extending generative replay to more complex architectures like Variational Autoencoders (VAEs) and applying the method to Reinforcement Learning to replace massive replay buffers in algorithms like DQN.
Summary updated