When a model trained on one data distribution (the "source") is deployed somewhere the inputs look different (the "target"), it usually has to be adapted using only unlabeled target data, because the source data cannot be shared for privacy or ownership reasons. This setting is called source-free domain adaptation. Adapting this way makes the model forget the source domain. The current fix is to synthesize fake source-style images, either by optimizing pixels to match statistics stored in the source network or by steering a diffusion model (a generator that turns random noise into images by repeated denoising). These images are then replayed during adaptation. The fix works, but image synthesis is the most expensive part of these pipelines. A separate literature on class-incremental learning has shown that forgetting can often be repaired far more cheaply. That repair tracks how the network's internal feature space moves and relocates stored class summaries to their new positions. No pixels are generated.
We propose to bring that feature-space repair into source-free adaptation, in a setting with several frozen source models and one compact student that must adapt to a sequence of target domains. The central observation is that source-free adaptation has a structural advantage that class-incremental learning lacks: the source models are frozen and remain available during adaptation. That availability lets the student re-fit a direct map from each teacher's feature space to its own current feature space at every step, rather than chaining step-to-step maps whose errors compound. Each target image is passed through all teachers and through the student to fit these maps. Each teacher's stored per-class feature statistics are then carried through its map, so the student can replay source classes as sampled features instead of synthesized images. The result is a single-backbone student that keeps multiple sources' knowledge while it adapts, at a small fraction of the cost of pixel-space replay.
Core Claim
We claim that, in continual multi-source-free adaptation, feature-level anchoring retains source-domain accuracy about as well as pixel-level synthetic replay, and costs orders of magnitude less compute. Feature-level anchoring means fitting teacher-to-student feature maps on unlabeled target images, transporting stored source class statistics through them, and replaying sampled features. We further claim that anchoring against frozen teachers avoids the error compounding over long task sequences that is a known open problem for drift compensation. The claim is falsifiable at two points. The first is whether a map fitted only on target-domain images still transports source-domain feature regions accurately. The second is whether the transported statistics actually protect source accuracy.
The first point is not obvious. Memory-Efficient Incremental Learning Through Feature Adaptation trains a two-layer network on current-class feature pairs from the old and new backbones. It finds that this network generalizes to old classes it never saw, and reports 84.3% on ImageNet-1000 with 488 MB of memory, against 63.7% and 3750 MB for rehearsal. That result, however, concerns unseen classes within one visual domain. Our setting asks whether the map generalizes across a domain gap, from target inputs to source inputs. The answer may be yes, because the maps relate two networks rather than two data distributions. It may also be no, if the student's features move differently on source-style inputs than on target inputs. Measuring this cleanly is the first scientific contribution.
Research Program
The setup has $M$ frozen source teachers with feature extractors $g_m$. Each teacher ships its classifier weights, its normalization statistics, and optionally a per-class Gaussian summary $(\mu_{m,k}, \Sigma_{m,k})$ of its source features. That summary is a few kilobytes per class and is in the spirit of the model specifications studied in A Study on PAVE Specification for Learnware, which release compact descriptors of a model's training distribution instead of the data. We also study a strictly weights-only variant, in which classifier weight rows act as class means and covariances are estimated from confident target features.
The student $f$ has one backbone. It adapts to target domains $T_1, \dots, T_S$ that arrive in sequence.
At each step the student is trained on target images with pseudo-labels from the teacher committee. We take the committee weighting directly from Agile Multi-Source-Free Domain Adaptation. That work (Bi-ATEN) freezes the source backbones and uses instance-level attention to weight each source's classifier and its trustworthiness per target image. We reuse this weighting as the labeling signal and do not claim novelty in pseudo-labeling.
The new component is the anchor. After each epoch we fit maps $A_m$ from teacher to student features on the current target batch:
$$ A_m = \arg\min_{A} \; \mathbb{E}_{x \sim T_s} \big\| A(g_m(x)) - f(x) \big\|^2 , $$
where $x$ ranges over unlabeled target images. The fit uses a stop-gradient on $f$, so that the map follows the student without restraining it. This follows the plasticity argument of Two-Way Is Better Than One (BiCyc), which trains paired forward and backward adapters with a cycle-consistency loss for class-incremental learning.
Source class $k$ from teacher $m$ is then replayed as features sampled from a Gaussian with mean $A_m(\mu_{m,k})$ and covariance $J_m \Sigma_{m,k} J_m^\top$. Here $J_m$ is the Jacobian of $A_m$ at the class mean, which equals $A_m$ itself when the map is linear. These replayed features supervise the student's classifier head, alongside the target loss, so that the decision boundaries for every source domain stay consistent with where those domains now sit in the student's feature space.
Because each $A_m$ always starts from a frozen teacher, the error at step $s$ does not inherit the error from step $s-1$. This is the property we will test against chained maps.
We compare three map families, which correspond directly to the curvature assumptions criticized in the drift literature. The first is a single linear map, as in the global linear drift of Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction (DPCR). DPCR models drift as a global linear shift plus per-class deformation via SVD and rebuilds the classifier in closed form. The second is the two-layer network of the feature-adaptation work above. The third is a cycle-consistent pair as in BiCyc, which pushes the map toward near-isometry and so preserves distances between classes. For the head we also test DPCR-style closed-form ridge regression on replayed features, which removes backpropagation from the anchoring step entirely.
Baselines
The naive reference point is the source-free student adapted with committee pseudo-labels and no source anchoring at all. This reference defines how much forgetting there is to fix. On PACS the continual source-free paper reports that unanchored adaptation drops source accuracy to 46.3%.
The strongest anchoring competitor is Continual Source-Free Unsupervised Domain Adaptation (C-SUDA). C-SUDA inverts the frozen source network's normalization statistics into source-style images, using confident target images as content priors, and co-trains on them with a frozen classifier head. It reports 78.0% source accuracy on PACS after continual adaptation. C-SUDA is single-source, so we run it once per teacher with the same student backbone, the same pseudo-labels, and the same target epochs. We also report its per-step GPU hours; the file notes this overhead but gives no figure, so we will measure it ourselves. Beating C-SUDA means matching its source retention within one point on PACS while using at least ten times less anchoring compute, and matching or exceeding its target accuracy.
The strongest generative competitor is Source-Free Domain Adaptation with Diffusion-Guided Source Data Generation (DM-SFDA). DM-SFDA fine-tunes Stable Diffusion against the source model's statistics to regenerate source images. It reports 79.5% average accuracy on Office-Home, 93.7% on Office-31, and 86.3% on VisDA-2017. Its compute cost is not reported in the file. Given our budget, we will run DM-SFDA only on a reduced Office-Home sequence, and otherwise compare against its published target numbers, clearly marked as from a different protocol.
For multi-source target accuracy, the reference is Bi-ATEN, which reports 59.6% on DomainNet with 10.6M trainable parameters and 369 samples per second. Bi-ATEN keeps all $M$ frozen backbones at inference, so its source retention is essentially perfect by construction. We treat it as an upper bound on retention and as the inference-cost reference, not as something to beat on accuracy. The relevant question is how much of Bi-ATEN's target accuracy a single-backbone student keeps, and at what inference cost.
For continual adaptation with teacher memory but no explicit anchoring, we run the dual-speed teacher scheme of SloMo-Fast. SloMo-Fast keeps a fast and a slow moving-average teacher and uses prototypes built on the fly from test data. It reports a 34.7% mean error on its continual test-time benchmarks.
Two ablations isolate the key ingredient. The first replays the stored source statistics without transporting them, the "stale prototype" condition; if it matches our method, drift compensation is unnecessary. The second chains step-to-step maps as in class-incremental learning instead of re-fitting from frozen teachers; if it matches, frozen-teacher anchoring is not the source of any gain. We will also include a Gaussian-kernel interpolation variant following Semantic Drift Compensation for Class-Incremental Learning, which estimates old-prototype drift from nearby current-sample displacements.
All methods share the backbone, the committee pseudo-labels, the target data order, and a hyperparameter search budget of the same number of trials, tuned on a held-out validation domain.
Why not just…?
Why not just recalibrate normalization statistics or minimize entropy at test time? Replacing normalization statistics with target statistics, as in Unsupervised BatchNorm Adaptation, and tuning normalization scale and shift by entropy minimization, as in Tent, are cheap and source-free. However, both overwrite exactly the statistics that encode the source domain. C-SUDA's 46.3% versus 78.0% gap on PACS shows how much source accuracy unanchored adaptation loses. Both methods also target a single model and give no way to fold several teachers into one student. We include both as baselines, so the experiments will show directly whether they close the gap.
Why not just keep the teacher ensemble and skip the student? This is a real threat. A frozen ensemble weighted Bi-ATEN-style forgets nothing and adapts well. Its cost is $M$ backbone forward passes per test image, forever. Our proposal is worthwhile only if a single student keeps most of the ensemble's target accuracy at roughly $1/M$ of the inference compute. We will report this trade-off explicitly as accuracy against inference FLOPs. If the student loses more than a few points to the ensemble, the honest conclusion is that ensembles should be kept wherever inference budget allows.
Why not just adapt the single best teacher? Confident Anchor-Induced Multi-Source Free Domain Adaptation (CAiDA) combines several source models through confident anchor points and a class-relationship consistency loss. It bounds target error more tightly with diverse sources and reports gains of 8.3 to 10.7 points over single-source-free methods on Digits-Five. We will nonetheless run a best-single-teacher-plus-anchoring condition, because for DomainNet domains with a single dominant source the advantage may shrink.
Why not just store the source features and replay them unchanged? This is the stale-prototype ablation described above. If the student's feature space barely moves on source-style inputs, it will win on simplicity. The diagnostic below measures how far the space moves, so the question is settled empirically rather than argued.
Why not just make image synthesis cheaper? NAYER speeds up data-free distillation 5 to 15 times by feeding text embeddings of class names to the generator, cutting CIFAR-100 to about two hours at 71.75% accuracy. It is in-domain and single-teacher, and class-name text carries no domain style, so it does not obviously produce source-domain-styled images. We would add a NAYER-style generator anchored per teacher only if feature anchoring fails the transport diagnostic below, since that failure is the case in which images would genuinely be needed.
Proposed Experiments
The main benchmark is continual multi-source DomainNet. Three domains serve as teachers, each with its own backbone, and the remaining three arrive as a target sequence, evaluated over all orderings. A shorter Office-Home version runs with one domain held out as the source-retention check. We use PACS in a single-source continual protocol for the direct comparison with C-SUDA's 78.0%.
We report target accuracy after each step, source accuracy on held-out source test sets (used only for evaluation), and average forgetting. On the cost side we report anchoring GPU hours per step and inference FLOPs per image. To stress error compounding, we extend the sequence by cycling target domains, as in SloMo-Fast's Cyclic-TTA idea, to at least twelve steps.
The decisive diagnostic uses held-out source images, at evaluation time only, to compute the true positions of source classes in the student's space. We then compare these with the transported means and covariances, measuring mean displacement error normalized by the distance between classes. Three questions follow from this diagnostic. Does transport error stay small despite the maps never seeing source inputs? Does it grow with step count for chained maps but not for frozen-teacher maps? And does it predict source accuracy across map families? If transport error is large and stale replay performs as well as transported replay, the core claim fails. If transported replay matches C-SUDA's retention at a small fraction of its cost, feature-level anchoring should replace pixel synthesis as the default for continual source-free adaptation.
References
- Memory-Efficient Incremental Learning Through Feature Adaptation
- A Study on PAVE Specification for Learnware
- Agile Multi-Source-Free Domain Adaptation
- Two-Way Is Better Than One: Bidirectional Alignment with Cycle Consistency for Exemplar-Free Class-Incremental Learning
- Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental Learning
- Continual Source-Free Unsupervised Domain Adaptation
- Source-Free Domain Adaptation with Diffusion-Guided Source Data Generation
- SloMo-Fast: Slow-Momentum and Fast-Adaptive Teachers for Source-Free Continual Test-Time Adaptation
- Semantic Drift Compensation for Class-Incremental Learning
- Unsupervised BatchNorm Adaptation (UBNA): A Domain Adaptation Method for Semantic Segmentation Without Using Source Domain Representations
- Tent: Fully Test-Time Adaptation by Entropy Minimization
- Confident Anchor-Induced Multi-Source Free Domain Adaptation
- Unleash Data Generation for Efficient and Effective Data-free Knowledge Distillation