Inherited Adapter Experts for Replay-Free Domain-Incremental Learning

Gyutae Oh, Hyunmin Kong, Jitae ShinarXiv 2026 · arXiv

TL;DR

Lamarck is a novel framework for replay-free domain-incremental learning that achieves state-of-the-art accuracy by strengthening per-domain experts rather than focusing on complex routing. By combining deep-block adapters with a simple "inheritance" mechanism—where each new expert is initialized from its predecessor—it significantly reduces the penalty of misrouting without requiring any stored images or extra parameters.

What Problem Does This Paper Address?

Domain-Incremental Learning (DIL) challenges a model to learn a fixed set of classes across a sequence of different domains (e.g., recognizing a "bicycle" first in clipart, then in sketches, then in real photos). In the "replay-free" setting, the model cannot store images from previous domains due to privacy or storage constraints, making catastrophic forgetting a major hurdle.

A popular strategy is to use per-domain experts: for every new domain, you freeze a pre-trained backbone (like a Vision Transformer) and train a small set of domain-specific parameters. At test time, a "router" must decide which expert should handle a given image. While recent research has focused on training sophisticated routers, this paper identifies a different bottleneck: the experts themselves aren't powerful enough, and when the router makes a mistake, the "wrong" expert is often completely lost.

Methods

The authors introduce Lamarck, a learner that combines three key components:

  1. Mahalanobis Router: Instead of a trained router, Lamarck uses a training-free approach based on Mahalanobis distance. It tracks the mean and covariance of frozen backbone features for each domain. At test time, an image is routed to the domain whose feature distribution it most closely matches.
  2. Adapter-Enhanced Experts: While many methods only use "prompts" (extra learnable tokens), Lamarck adds adapters—small residual bottleneck layers—after every deep block (blocks 6–12) of the frozen ViT. These adapters modify every image token, providing a more powerful way to correct domain-specific features.
  3. Expert Inheritance: This is the "Lamarckian" twist. Instead of training each expert from scratch, expert $t$ is initialized with the parameters of expert $t-1$. Since all domains share the same classes, this inheritance ensures that even if an image is misrouted to a later expert, that expert still retains useful classification knowledge from its ancestors.
Figure 2. Lamarck. (a) One expert on the frozen backbone. Prompts retrieved from the expert’s prompt pool enter the early blocks, and after each deep block a residual adapter corrects every image token. An MLP readout on the final CLS feature computes the logits. (b) Expert t is initialized from expert t−1, trained on domain t and then frozen. (c) A training-free router selects the domain whose mean feature is nearest to the frozen-ViT feature z(x) in shrunk Mahalanobis distance, and only that domain’s expert classifies x.

Datasets

The model was evaluated on three diverse benchmarks:

  • DomainNet (DN): Six domains ranging from clipart to sketch, with 345 classes.
  • OfficeHome (OH): Four domains (Art, Clipart, Product, Real World) with 65 classes.
  • CDDB-Hard: A challenging deepfake detection benchmark featuring five different generation sources (e.g., GauGAN, BigGAN).

Key Insights and Ideas

  • Routing isn't the bottleneck: The authors show that even with a perfect router, existing methods are limited by the accuracy of the experts themselves. The "routing loss" (the gap between perfect routing and their simple Mahalanobis router) is surprisingly small—under 2.7 points.
  • The power of inheritance: Initializing from a predecessor is "free" (no extra parameters or compute) but highly effective. It makes later experts more accurate on earlier domains. If the router accidentally sends a "clipart" image to the "photo" expert, the photo expert is much more likely to get it right if it was initialized from the clipart expert.
  • Adapters > Prompts in deep blocks: While prompts work well in early layers to guide attention, adapters are more effective in deeper layers for transforming features. Adding adapters raised accuracy by 4.4 points on the CDDB-Hard benchmark compared to a prompt-only version.
Inheritance reduces the cost of misrouting to later experts.

Key Results

  • State-of-the-Art Performance: Lamarck achieved the highest mean accuracy (83.62%) across the three benchmarks, outperforming twelve other frozen-backbone baselines.
  • Deepfake Detection Lead: It particularly excelled on CDDB-Hard, leading the next best baseline by 5.2 accuracy points.
  • Efficiency: Lamarck maintains a competitive inference cost. Like other "two-pass" learners, it runs the backbone once to route and once to classify, sitting in the middle of the pack for GFLOPs while delivering superior accuracy.
Figure 3. Lamarck has the highest mean ACC at the inference cost of a two-pass learner. Mean ACC over the three benchmarks (Tab. 2) against GFLOPs per image on DN (log scale). Throughput in Tab. S9 falls as GFLOPs rise (rank correlation −0.94). SOYO costs most: its released inference runs the ViT once per domain.

Limitations and Future Work

  • Storage Scaling: Because Lamarck stores one set of adapters and a classification head per domain, its storage requirements grow linearly with the number of domains. While these parameters are small (~0.34M per expert), this could become an issue for scenarios with hundreds of domains.
  • Domain-Specific Gaps: On the DomainNet benchmark, another method (SOYO) still holds a slight lead in own-domain accuracy for certain domains like sketches.
  • Future Directions: The authors suggest that exploring ways to share or compress these expert parameters further without losing the benefits of inheritance could be a fruitful next step.

Summary updated

Abstract

Replay-free domain-incremental learning trains a model on the same classes in successive domains without keeping earlier images. Per-domain learners freeze a pre-trained ViT and train an additional expert for each new domain: the small set of parameters and the classification head learned on that domain. Test images carry no domain label, so a router selects one expert for each image. Recent work trains this router. However, a training-free Mahalanobis router costs the analyzed learners under 2.7 points of final accuracy relative to always selecting each image's own-domain expert. The experts, not this router, limit accuracy. Our learner, Lamarck, therefore uses this router and strengthens the experts. Its adapters, small trainable layers that correct every image token after each deep ViT block, add 4.4 points of final accuracy on the deepfake benchmark CDDB-Hard. In addition, each expert is initialized from its predecessor at no extra cost: the domains share their classes, so an image misrouted to a later expert loses less. Against twelve frozen-backbone baselines under one protocol, Lamarck is first on CDDB-Hard by 5.2 points, second on DomainNet and OfficeHome, and highest in mean accuracy.
BibTeX
@misc{oh2026inherited,
  author = {Gyutae Oh and Hyunmin Kong and Jitae Shin},
  title = {{Inherited Adapter Experts for Replay-Free Domain-Incremental Learning}},
  year = {2026},
  eprint = {2611.24145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24145},
}
Figure