TL;DR
Lamarck is a novel framework for replay-free domain-incremental learning that achieves state-of-the-art accuracy by strengthening per-domain experts rather than focusing on complex routing. By combining deep-block adapters with a simple "inheritance" mechanism—where each new expert is initialized from its predecessor—it significantly reduces the penalty of misrouting without requiring any stored images or extra parameters.
What Problem Does This Paper Address?
Domain-Incremental Learning (DIL) challenges a model to learn a fixed set of classes across a sequence of different domains (e.g., recognizing a "bicycle" first in clipart, then in sketches, then in real photos). In the "replay-free" setting, the model cannot store images from previous domains due to privacy or storage constraints, making catastrophic forgetting a major hurdle.
A popular strategy is to use per-domain experts: for every new domain, you freeze a pre-trained backbone (like a Vision Transformer) and train a small set of domain-specific parameters. At test time, a "router" must decide which expert should handle a given image. While recent research has focused on training sophisticated routers, this paper identifies a different bottleneck: the experts themselves aren't powerful enough, and when the router makes a mistake, the "wrong" expert is often completely lost.
Methods
The authors introduce Lamarck, a learner that combines three key components:
- Mahalanobis Router: Instead of a trained router, Lamarck uses a training-free approach based on Mahalanobis distance. It tracks the mean and covariance of frozen backbone features for each domain. At test time, an image is routed to the domain whose feature distribution it most closely matches.
- Adapter-Enhanced Experts: While many methods only use "prompts" (extra learnable tokens), Lamarck adds adapters—small residual bottleneck layers—after every deep block (blocks 6–12) of the frozen ViT. These adapters modify every image token, providing a more powerful way to correct domain-specific features.
- Expert Inheritance: This is the "Lamarckian" twist. Instead of training each expert from scratch, expert $t$ is initialized with the parameters of expert $t-1$. Since all domains share the same classes, this inheritance ensures that even if an image is misrouted to a later expert, that expert still retains useful classification knowledge from its ancestors.
Datasets
The model was evaluated on three diverse benchmarks:
- DomainNet (DN): Six domains ranging from clipart to sketch, with 345 classes.
- OfficeHome (OH): Four domains (Art, Clipart, Product, Real World) with 65 classes.
- CDDB-Hard: A challenging deepfake detection benchmark featuring five different generation sources (e.g., GauGAN, BigGAN).
Key Insights and Ideas
- Routing isn't the bottleneck: The authors show that even with a perfect router, existing methods are limited by the accuracy of the experts themselves. The "routing loss" (the gap between perfect routing and their simple Mahalanobis router) is surprisingly small—under 2.7 points.
- The power of inheritance: Initializing from a predecessor is "free" (no extra parameters or compute) but highly effective. It makes later experts more accurate on earlier domains. If the router accidentally sends a "clipart" image to the "photo" expert, the photo expert is much more likely to get it right if it was initialized from the clipart expert.
- Adapters > Prompts in deep blocks: While prompts work well in early layers to guide attention, adapters are more effective in deeper layers for transforming features. Adding adapters raised accuracy by 4.4 points on the CDDB-Hard benchmark compared to a prompt-only version.
Key Results
- State-of-the-Art Performance: Lamarck achieved the highest mean accuracy (83.62%) across the three benchmarks, outperforming twelve other frozen-backbone baselines.
- Deepfake Detection Lead: It particularly excelled on CDDB-Hard, leading the next best baseline by 5.2 accuracy points.
- Efficiency: Lamarck maintains a competitive inference cost. Like other "two-pass" learners, it runs the backbone once to route and once to classify, sitting in the middle of the pack for GFLOPs while delivering superior accuracy.
Limitations and Future Work
- Storage Scaling: Because Lamarck stores one set of adapters and a classification head per domain, its storage requirements grow linearly with the number of domains. While these parameters are small (~0.34M per expert), this could become an issue for scenarios with hundreds of domains.
- Domain-Specific Gaps: On the DomainNet benchmark, another method (SOYO) still holds a slight lead in own-domain accuracy for certain domains like sketches.
- Future Directions: The authors suggest that exploring ways to share or compress these expert parameters further without losing the benefits of inheritance could be a fruitful next step.
Summary updated