PatchAdaNet: Unsupervised class incremental learning using local patches and global adapters

Kamel Aouaidjia, Yanchao Duan, Zihui Zhen, Xin Li, Yongfei Lv, Chongsheng ZhangarXiv 2026 · arXiv

TL;DR

PatchAdaNet is a novel, unsupervised framework for class-incremental learning that eliminates "catastrophic forgetting" without needing a memory buffer to store past examples. By combining frozen, patch-level feature extractors with lightweight learnable adapters and a "Global Feature Centroid" (GFC) to anchor representations, it matches or outperforms many state-of-the-art methods that rely on replaying old data.

What Problem Does This Paper Address?

Deep learning models are notoriously bad at "continual learning." When trained on a sequence of new tasks, they tend to overwrite what they learned previously—a phenomenon known as catastrophic forgetting.

Most current solutions fall into three buckets, each with a catch:

  1. Replay-based methods: They store a "memory buffer" of old images. This works well but raises privacy concerns, and the buffer grows as you see more tasks.
  2. Architectural expansion: They add new neurons for every task. This makes the model size explode over time.
  3. Regularization: They penalize changes to important weights. This often fails to isolate task-specific knowledge from shared features, leading to interference.

The authors of PatchAdaNet aim for a "best of all worlds" approach: a model that is replay-free (no stored images), fixed-size (no parameter growth), and unsupervised (doesn't need labels during feature learning).

Challenges of existing methods vs. PatchAdaNet solutions

Methods

PatchAdaNet uses a clever two-stage design to separate "universal" knowledge from "task-specific" knowledge.

1. The Frozen Foundation (Stage 1)

The researchers observed that all images share basic structural building blocks like edges, corners, and textures. They divide images into a grid of patches and train independent autoencoders (either CNN or Vision Transformer based) to reconstruct these patches. Once trained on an external dataset, these patch encoders are frozen permanently. Because they never update during the actual incremental learning phase, they are "immune to forgetting" by construction.

Illustration of patch partitioning and training

2. Global Adapters and the GFC (Stage 2)

During the actual learning of new tasks, the model aggregates these frozen patch features into a global image representation. On top of these, they add learnable adapter layers. To keep these adapters from drifting too far and forgetting old tasks, they introduce two mechanisms:

  • Elastic Weight Consolidation (EWC): A standard technique that penalizes changes to weights important for prior tasks.
  • Global Feature Centroid (GFC): This is the paper's secret sauce. It maintains a running average of the adapter outputs. During training on a new task, the model is penalized if its current features drift too far from this "centroid" of previous knowledge. This acts like a "virtual replay" of features without actually storing any data.
Detailed illustration of the PatchAdaNet training process

Datasets

The authors evaluated PatchAdaNet on several standard benchmarks:

  • CIFAR-10 & CIFAR-100: Classic image classification sets split into 5 and 20 tasks, respectively.
  • miniImageNet: A more complex dataset split into 20 tasks.
  • 5-Datasets: A sequence of five different domains (MNIST, SVHN, notMNIST, Fashion-MNIST, and CIFAR-10) to test robustness to extreme domain shifts.
  • TinyImageNet: Used only for the initial "Stage 1" pretraining of the patch encoders.

Key Insights and Ideas

  • Modularity is Key: By isolating low-level feature extraction (frozen patches) from high-level integration (adapters), the model creates a stable base that doesn't need to be relearned.
  • Feature Anchoring > Data Replay: The GFC proves that you don't need to store raw pixels to prevent representation drift. A simple running average of feature outputs can serve as an effective anchor.
  • ViT Superiority: One of the most interesting findings is that the Vision Transformer (ViT) variant of PatchAdaNet is significantly more robust than the CNN version. ViTs were much less sensitive to the order in which tasks were presented and handled domain shifts (like moving from grayscale digits to color cars) much more gracefully.

Key Results

PatchAdaNet delivers impressive performance for a replay-free method:

  • CIFAR-10: Achieved 65.9% accuracy, significantly outperforming the replay-based baseline OnPro (57.8%).
  • CIFAR-100: The ViT variant reached 32.0%, which is higher than many supervised methods that actually use a memory buffer.
  • Unsupervised Success: Even though the features are learned without labels (via reconstruction), a simple linear classifier trained on top of the resulting features achieves competitive results against fully supervised models.

Limitations and Future Work

While powerful, the authors note a few areas for improvement:

  • Model Scale: The experiments used relatively compact models. It remains to be seen if these benefits scale to massive backbones like ViT-Huge.
  • GFC Noise: The GFC is currently updated using only the final mini-batch of a task. This could introduce noise if that specific batch isn't representative of the whole task.
  • Pretraining Dependency: The quality of the "frozen foundation" depends on the diversity of the pretraining data (TinyImageNet). Using even larger, more diverse datasets for Stage 1 could further improve generalization.

The researchers suggest that future work should explore replacing the lightweight patch autoencoders with larger, pretrained "foundation models" to see if the adapter/GFC approach can unlock even higher performance in open-ended learning scenarios.

Summary updated

Abstract

Biological learning systems acquire knowledge incrementally through broad, reusable representations established early and kept largely stable, while integrating new experience with minimal interference with previously learned knowledge. Replay-based methods approximate this by storing samples or features from previous tasks and mixing them into the current task. However, memory buffers grow with the number of tasks and may retain task-specific noise that reduces generalization. To address these limitations, we propose PatchAdaNet (Patch Adapter Network), an unsupervised class-incremental learning method for image classification that continually updates feature representations using autoencoders. Since images share structural features such as edges, corners, and shapes, we first divide each image into patches and train independent autoencoders on external data to reconstruct each patch. The layers of all patch encoders are aggregated into global frozen layers that generalize across tasks and remain fixed throughout learning. Learnable adapters then build global image representations from these features to reconstruct the entire image. To prevent feature drift without storing data, we introduce the Global Feature Centroid (GFC), a fixed-size running average of adapter outputs updated after each task to penalizes representation drift. Elastic Weight Consolidation (EWC) is applied to the learnable layers to preserve previously learned features by constraining important weights. Finally, a linear classifier is trained on the frozen latent representation and GFC for class-incremental evaluation. We evaluate PatchAdaNet in both CNN and Vision Transformer settings on CIFAR-10, CIFAR-100, miniImageNet and 5-Datasets, providing a comparative study of the two architectures under identical class-incremental settings. Code is available.
BibTeX
@misc{aouaidjia2026patchadanet,
  author = {Kamel Aouaidjia and Yanchao Duan and Zihui Zhen and Xin Li and Yongfei Lv and Chongsheng Zhang},
  title = {{PatchAdaNet: Unsupervised class incremental learning using local patches and global adapters}},
  year = {2026},
  eprint = {2611.24137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24137},
}
Figure