In deep learning, "catastrophic forgetting" occurs when a model trained on a new task overwrites the weights essential for previous tasks. To prevent this, traditional methods often use rehearsal: storing a subset of raw images from old tasks and replaying them during new training sessions. However, storing raw data is computationally expensive, memory-intensive, and often prohibited by privacy regulations.
A growing body of research suggests a more elegant solution: instead of storing raw data, models should store prototypes—mathematical summaries of each category in the form of mean feature vectors. By treating these prototypes as stable anchors in a high-dimensional embedding space, models can learn new information without losing their grip on the past.
The Prototype as a Compressed Memory
At its core, a prototype is the "central tendency" of a class. Rather than remembering every specific bird or car the model has seen, it stores a single vector representing the average features of that category. Weijian Deng and colleagues in their work on Categorical Memory Networks demonstrate that these prototypes can be updated during training using a moving average. This creates a "categorical memory" that captures how a specific image relates to the broader taxonomy.
This approach is remarkably efficient. Biplab Banerjee and his team found that representing historical knowledge through class-level feature vectors requires approximately 38% less memory than traditional rehearsal-based methods while maintaining competitive accuracy.
Anchoring the Embedding Space
Why does storing a single vector prevent a complex neural network from forgetting? The intuition lies in the stability-plasticity dilemma. A model needs "plasticity" to learn new things but "stability" to keep old ones.
Prototypes act as gravitational anchors. In the Prior Knowledge-Infused (PKI) network, Kexin Bao and Dan Zeng show that replaying these mathematical representations alongside new data prevents the model's feature extractor from "drifting." Similarly, Fahad Sarfraz and Elahe Arani argue in their Semantic-Aware Representation Learning (SARL) framework that prototypes facilitate similarity-based grouping, ensuring that new tasks are learned in a way that respects the semantic boundaries of old ones.
Mitigating Noise and Concept Drift
A single prototype can sometimes be "noisy" if it is based on a few unrepresentative samples. To solve this, researchers like Chen Chen and Jing Wang have proposed momentum-updated banks. By updating prototypes slowly over time, the model builds a robust, aggregate representation that is less sensitive to outliers in individual training images.
This stability is crucial when data distributions shift over time, a problem known as "concept drift." In Deep Streaming View Clustering, Honglin Yuan uses a dynamic knowledge base of prototypes to align new, incoming perspectives with historical context, ensuring that clusters remain separable even as the data evolves.
Privacy-Preserving Learning
Perhaps the most significant advantage of prototypes is their inherent privacy. In security-sensitive applications like person re-identification, storing raw images of individuals is often a violation of privacy.
Jiahuan Zhou and Yuxin Peng address this in their CKDA framework, which matches people across visible and infrared cameras. By using mathematical feature prototypes instead of raw data replay, they mitigate catastrophic forgetting without ever needing to store private user images. This same principle is applied by Binbin Yang in continual object detection, where prototypes serve as a lightweight, privacy-preserving alternative to exemplars.
Beyond Simple Classification
The utility of prototypes extends beyond simple image labels:
- Object Detection: Qirui Wu and colleagues found that forgetting in detectors is often localized in the classifier. Their NSGP-RePRE framework uses "regional prototypes" to capture both the semantic center and the intra-class variance (like different poses) of objects.
- Natural Language Processing: In Federated Incremental Named Entity Recognition, Chenxing Li uses prototype-guided distillation to help decentralized clients learn new entity types without sharing sensitive text data.
- Data Synthesis: In the I²CANSAY framework, SongLin Dong and Yuhang He take prototypes a step further, using them as the foundation to synthesize "pseudo-features" for training, allowing the model to "re-imagine" old classes without ever seeing the original images again.