Drift-Priced Routing for Continually Updated Adapters

Personalized language models are increasingly served as one shared base model plus many small per-user or per-cohort add-on weight sets called LoRA adapters (low-rank adapters, a few megabytes of extra weights that specialize the base model). Once those adapters are retrained continually to follow users whose tastes change, a scheduling problem appears that neither the load-balancing literature nor the preference-drift literature has addressed. After each update, the servers in a cluster hold copies of a user's adapter at different ages. Sending the request to the server with the newest copy concentrates load on that server. Sending it to an idle server means either loading the newest weights there, which is slow, or serving an older copy, which may give worse answers. The central observation of this proposal is that the cost of serving an old copy is not a function of its age. It depends on how much this particular user's model has actually changed since that copy was made. That quantity is cheap to measure at training time. For users whose preferences are stable, most servers are acceptable destinations. For users in the middle of a preference shift, only freshly updated servers are.

We propose a dispatcher that treats "how far has this adapter moved since the resident copy" as a price in the routing decision, alongside queue delay and weight-loading time. The effect is to widen the load-balancing choice set for stable users and narrow it for drifting ones. We will build this on top of multi-adapter serving kernels. We will evaluate it on workloads whose adapters are trained with existing continual-preference methods on chronologically ordered data. The claim is that, at matched personalization quality, drift-priced routing sustains substantially more load under a latency target than routing that always demands the newest adapter. It should also beat routing that bounds staleness by age alone, but only when drift rates differ across users. We will test that condition directly, and the claim stands or falls on it.

Core Claim

The two directions play different roles here. Task distribution efficiency supplies the mechanism: choosing among a few candidate servers, as in the "power of $d$ choices" rule, where sending a job to the less loaded of two random servers collapses the maximum queue from $O(\log n/\log\log n)$ to $O(\log\log n)$. Temporal preference drift supplies the regime and the key signal. When users' preferences move at heterogeneous and time-varying rates, the set of servers that can acceptably answer a request is user-dependent and shifts over time.

Concretely, when a request from user $u$ arrives, the dispatcher samples a small candidate set of servers $C$. It always includes any server holding some version of $u$'s adapter. It then routes to

$$ s^\star = \arg\min_{s \in C} \; \hat W_s + L_s(u) + \beta \, D_u(v^\ast, v_s), $$

where $\hat W_s$ is the estimated queueing wait at server $s$, $L_s(u)$ is the time to load $u$'s newest adapter onto $s$ (zero if $s$ already holds it, or if we choose to serve its resident copy), $v^\ast$ is $u$'s newest adapter version, $v_s$ is the version resident on $s$, $D_u$ is a divergence between the two versions' output distributions, and $\beta$ converts quality loss into latency units. An equivalent constrained form, which is easier to set from a service agreement, serves the resident copy only when $D_u(v^\ast, v_s) \le \varepsilon$ and otherwise charges the load time. The new element is $D_u$. It is computed once per adapter update on a small fixed probe set of prompts, so routing costs no more per request than a lookup.

This choice of signal is motivated by Continual Low-Rank Adapters for LLM-based Generative Recommender Systems. That paper's PESO method updates a single evolving adapter under a softmax-KL proximal penalty and shows that this penalty acts as a gate: it admits updates only along directions with strong new evidence. As a consequence, per-update movement is small for stable users and large for drifting ones. The proximal term already measures the divergence we need, so the training side exports $D_u$ nearly for free.

Where This Sits

The closest work is DualMap. It hashes each prompt prefix to two candidate servers and uses the one holding the cached attention state (the KV cache, stored intermediate results that let a server skip recomputing a shared prompt prefix). It falls back to the second server only when the first would violate a time-to-first-token (TTFT) target. DualMap's notion of cache content is binary: a server either has the exact prefix or it does not, and the content never becomes partially valid. Our setting has graded validity: a resident adapter can be slightly or badly out of date. DualMap has no way to express that.

The multi-adapter serving systems S-LoRA, Punica and CLIMB supply the method layer we reuse: batched kernels, unified paging of adapter weights, and CLIMB's attention to the performance cliff when adapters no longer fit in GPU memory. All three treat adapters as frozen artifacts, which our proposal drops.

On the drift side, Right Now, Wrong Then (NS-DPO, which downweights old preference pairs by $\gamma^{T-t_i-1}$) and PESO show that tracking drift requires updating models over time. That is what creates the version-skew problem. Neither says anything about serving. A Shared Low-Rank Adaptation Approach to Personalized RLHF supplies the per-user adapter structure, a shared up-projection with small per-user factors, that makes many concurrently evolving adapters realistic.

Two load-balancing results constrain the design. Balanced allocations with heterogenous bins proves that when some bins are sampled far more often, $d$ must grow to keep them from becoming bottlenecks. In our terms, a drifting user has an effective $d$ close to 1, so hotspot risk concentrates on exactly those users, and our design must spend replication on them. Revisiting Consistent Hashing with Bounded Loads shows that local overflow to hash-ring neighbors cascades: about 35,000 cache misses versus 312 under random rehashing on the AOL log. When a drifting user's fresh server saturates, spillover should therefore go to random servers rather than neighboring ones. Load is not what you should balance: Introducing Prequal supplies our wait estimator $\hat W_s$: asynchronous probes of requests-in-flight and latency. Prequal's own result is that balancing CPU load is the wrong target for tail latency.

Research Program

The system has three parts. The first is the training side, which runs continual preference updates per user or per cohort. On each update it emits a new version together with $D_u$ against the last few versions still resident anywhere in the cluster. $D_u$ is measured as mean token-level KL divergence on a fixed probe set of a few hundred prompts, or read from PESO's proximal term when PESO is the trainer.

The second is the dispatcher, implementing the rule above with Prequal-style probing. Two additions come from the load-balancing literature. First, when a drifting user's fresh server exceeds its wait budget, the spillover target is chosen by random jump, following RJ-CH. Second, a pull-based replication path lets idle servers pull the newest versions of the currently highest-drift, highest-traffic adapters. This follows the observation in Delay, Memory, and Messaging Tradeoffs in Distributed Service Systems that letting idle servers pull work achieves low delay with little messaging in heavy traffic.

The third part is analysis. We will model the system as balls-into-bins in which ball $u$ has an eligible set whose size shrinks with its current drift. We will derive how the maximum load depends on the traffic-weighted fraction of high-drift users, extending the heterogeneous-bins analysis. The qualitative prediction, which follows from that paper's requirement that $d$ scale with sampling skew, is that the gain over always-newest routing grows with the share of traffic from stable users and vanishes when all users drift quickly.

The workloads pair real drift with realistic arrivals. For drift, we will train PESO-style evolving adapters for several hundred user cohorts on the chronologically partitioned Amazon Review 2023 and Yelp streams used by PESO. We will also train NS-DPO adapters for many synthetic tenants on TV-HH and the label-flipped UltraFeedback variants, each tenant with its own change-point schedule, so that the ground-truth drift timing is known. For arrivals, we will map sessions from the Mooncake conversation and Tool&Agent traces used by DualMap onto these tenants, on 8 to 32 GPU instances of a 7B model. Quality is measured per request against the version that was actually served. For Amazon and Yelp this is Hit@5/10 and NDCG@5/10 on the held-out next interaction. For NS-DPO tenants it is reward accuracy on the current regime. Serving is measured by P90 and P99 TTFT and by effective capacity, meaning the highest arrival rate that meets a 5-second TTFT target, which is DualMap's definition.

Baselines

The naive reference is random assignment, with Prequal-style latency-aware selection as the strong load-only variant. Both always load the newest adapter wherever the request lands. They show what pure load balancing achieves when affinity is ignored and loading is paid in full. Strict affinity, meaning each user is pinned to one server that always holds the newest version, is the opposite extreme. It exposes the hotspot failure that DualMap documents.

The strongest existing method is DualMap run with the adapter identifier as the hash key and strong consistency, meaning only newest-version copies count as hits. On its own traces, DualMap reports effective capacity under a 5-second TTFT limit improved by up to 2.25× and high-traffic P90 TTFT reduced by 82.3% to 97% relative to prior methods. No reported result was found on Lacuna for DualMap or any router on evolving-adapter workloads. We will therefore reproduce DualMap on our workloads using the same base model, GPU count, adapter paging layer, arrival traces, and a tuning budget matched to ours for its hotness-tree and SLO parameters. Beating it means higher effective capacity at the 5-second target while keeping per-request quality within the seed-to-seed variation of the always-newest reference, both averaged over at least three trace replays.

The quality ceiling for every method is "always newest version," and the quality floor is "never update," a frozen adapter. PESO's reported 3.7% to 6.3% relative ranking gains over its competitors and NS-DPO's retained reward accuracy above 60–70% versus 50–55% for standard DPO after preference flips calibrate how large the ceiling-to-floor gap should be. If our reproductions show a much smaller gap, staleness is nearly free and the whole problem shrinks.

Two ablations isolate the key ingredient. Age-bounded staleness serves a resident copy if it is younger than a tuned age threshold, which removes the drift signal while keeping everything else. Periodic synchronization pushes new versions to all replicas every $K$ minutes and routes freely between pushes. Both receive the same tuning budget as our $\beta$ or $\varepsilon$.

Why not just…?

Why not just use DualMap with adapter IDs as keys? DualMap handles skew in whether a server holds the content, but it treats any version mismatch as a total miss. Every adapter update therefore invalidates the hot servers and forces reloads exactly when a drifting user is active. That is the burst the continual-learning side predicts, because drift events trigger both retraining and user activity. The DualMap-with-strong-consistency baseline measures this directly. If it matches our router, the lesson is that graded validity does not matter at realistic update rates.

Why not just bound staleness by version age? This is the real threat. If every user drifts at roughly the same rate, age is a good proxy for divergence and a tuned age threshold will do as well as our router. Our claim holds only if divergence is heterogeneous across users at equal age. PESO's gate, which moves stable users' adapters little, suggests it is, but we have not measured it. The first decisive experiment measures it before anything else is built, and the age-bounded ablation settles the question end to end.

Why not just make adapter loading fast enough that affinity stops mattering? S-LoRA's unified paging and Punica's batched kernels reduce the cost of switching adapters. If loading is cheap relative to queueing, every server can load the newest version, and the staleness trade-off disappears. CLIMB's framing around a residency cliff suggests the cost returns sharply once adapters exceed GPU memory, but the file reports no figure for its size. We will sweep the number of active adapters from well below to well above resident capacity. We predict negligible gains below the cliff and large gains above it. If the gains are negligible everywhere, we will say so.

Why not just batch updates and sync periodically? Periodic sync bounds staleness uniformly. It wastes transfers on stable users and under-serves users in mid-shift, whose quality drops between pushes. The periodic-sync ablation, compared at equal network transfer volume, tests whether this matters in practice.

Proposed Experiments

The first decisive check needs no serving system. On the Amazon and Yelp PESO cohorts and on the NS-DPO tenants, we will record, for every pair of versions, their age gap, their probe-set divergence $D_u$, and the quality drop from serving the older version. The claim requires two things: $D_u$ must predict quality drop substantially better than age does, and the spread of $D_u$ across users at fixed age must be large. If age predicts quality drop as well as $D_u$, we will abandon the drift price and report age-bounded routing as sufficient.

The second check runs the full trace-driven cluster comparison against all baselines. We will vary three factors: the fraction of tenants in active drift, controlled through NS-DPO change points; update frequency; and adapter count relative to GPU residency. The analysis predicts that the capacity gain over DualMap-with-strong-consistency rises with the share of stable-user traffic and above the residency cliff, and disappears when all tenants drift together. We also expect the random-jump spillover to show fewer cascaded tail-latency spikes than neighbor spillover during synchronized drift events. That would be the analogue in this setting of the CH-BL versus RJ-CH gap.

The third check tests robustness when the divergence price is wrong. We will perturb $D_u$ with noise and with deliberate lag, simulating delayed training jobs, and measure how quickly quality degrades. The goal is to establish how accurate and how timely the training side must be before the router's advantage disappears.