Analysis of the Memorization and Generalization Capabilities of AI Agents: Are Continual Learners Robust?

Minsu Kim, Walid Saad

Abstract

Continual Learning (CL) typically focuses on mitigating catastrophic forgetting of past tasks. However, real-world deployment requires agents to be robust to unseen environmental shifts. This paper introduces a novel CL framework based on Probabilistic Domain Generalization (PDG) that optimizes for a "probabilistic worst-case" risk. We demonstrate a fundamental tradeoff: while larger memory buffers enhance an agent's robustness to change, they simultaneously increase the difficulty of perfectly mastering all individual past experiences. Experimental results on Rotated MNIST show that our approach significantly outperforms existing baselines in generalization accuracy.

Introduction

In dynamic environments, AI agents encounter a continuous stream of data characterized by non-stationary distributions. A self-driving car, for instance, must adapt from sunny conditions to snowy ones without losing its previous capabilities. While most CL research addresses catastrophic forgetting, the ability to generalize to entirely new environments—robustness—is equally critical.

Current methods often rely on explicit task identities, which are rarely available in practice. This research tackles the dual challenge of achieving high memorization (performance on past data) and robust generalization (performance on future, unseen data) in a general CL setting where task boundaries are unknown.

Methodology

The proposed framework treats environmental risks as random variables. Instead of minimizing average loss, the objective is to minimize the worst-case risk across potential environments.

Probabilistic Domain Generalization (PDG)

The core components of the framework include:

  • Capacity-Limited Memory: A buffer stores a representative subset of environmental data encountered during training.
  • Risk Distribution Estimation: The agent samples multiple batches from its memory to estimate the mean and variance of risks across different environments.
  • Gaussian Approximation: Risks are modeled using a Gaussian distribution. This allows the calculation of a performance "quantile" based on a parameter $\alpha$, providing a probabilistic guarantee of robustness.
  • Balanced Objective: The final loss function integrates empirical risk on past data (for memorization) with the estimated worst-case risk (for generalization).

Experimental Setup

The framework was evaluated using the Rotated MNIST dataset, where digits are rotated from 0° to 150° in 25° increments. Each rotation represents a distinct distributional shift. To test generalization, specific rotations (e.g., 0° or 150°) were held out during training as "target" tasks.

The Memory Size Tradeoff

A key contribution of this work is the mathematical and empirical demonstration of the memory size tradeoff. As memory size $|\mathcal{M}|$ increases:

  1. Generalization Improves: A larger buffer provides a more diverse sample of the world, reducing estimation bias and making the agent more robust to unseen shifts.
  2. Memorization Becomes Harder: Finding a single set of model parameters that is optimal for a larger, more diverse set of stored environments becomes increasingly difficult.
Tradeoff between memorization and generalization across different memory sizes.

Results and Discussion

The proposed algorithm consistently outperformed popular baselines, including Experience Replay (ER), Dark Experience Replay (DER++), and Elastic Weight Consolidation (EWC).

Performance comparison of the proposed method against standard CL baselines on Rotated MNIST.

On the most challenging unseen environments (0° and 150° rotations), the framework achieved up to a 10% gain in generalization accuracy over standard memory-based baselines. These results validate the theoretical insight that optimizing for probabilistic worst-case risk leads to more robust continual learners.

Limitations and Future Work

While effective, the current approach relies on a Gaussian assumption for risk distributions, which may not hold for highly complex, multi-modal environments. Additionally, the requirement to sample multiple batches from memory increases computational overhead. Future research will explore more complex density estimators and modular architectures to scale these robust learning principles to high-dimensional robotics and real-world applications.

Summary updated

BibTeX
@inproceedings{kim2024analysis,
  author = {Minsu Kim and Walid Saad},
  title = {{Analysis of the Memorization and Generalization Capabilities of AI Agents: Are Continual Learners Robust?}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10447575},
}
Figure