Rank-Elastic LoRA Adapters for Load-Adaptive Multi-Tenant Serving

Many companies now serve thousands of fine-tuned variants of one large language model. Each variant is stored as a small add-on, a LoRA adapter (a low-rank correction $BA$ added to each frozen weight matrix, where the rank $r$ sets how many directions the correction can use). Serving systems keep one copy of the base model on the GPU and swap adapters in and out. Two costs grow with each adapter's rank. The first is memory and host-to-GPU transfer. The second is per-step memory bandwidth when a decoding batch mixes many different adapters, because the base weights are read once per step but every distinct adapter's weights must be read separately. A rough estimate: a rank-64 adapter on every projection of a 7B model is about 160M parameters, or roughly 320 MB in 16-bit precision. A batch touching a few dozen distinct adapters then reads about as many adapter bytes as base-model bytes. Today the rank is fixed when the adapter is trained. When load spikes, the server can only queue requests, evict adapters, or drop them.

We propose making adapter fidelity a dial the server can turn at run time. Each adapter is trained so that every prefix of its components, ordered from most to least important, is itself a usable adapter. Quality then degrades smoothly and predictably as components are dropped. A lightweight scheduler measures each adapter's quality-versus-size curve offline. Under memory or bandwidth pressure, it chooses how many components of each resident adapter to load, so that total expected quality is as high as the budget allows. In plain terms, the server can hand an adapter "80% of itself" instead of all of it or none of it, and it knows in advance what that costs in accuracy.

Core Claim

The claim is that load-driven, per-adapter truncation beats both of today's options for a GPU with a fixed memory and bandwidth budget. One option is fixed-rank serving with paging. The other is uniformly small ranks. The proposal wins if two conditions hold. First, each adapter is trained so that its component prefixes form an ordered family of approximations. Second, the truncation order is global across layers, not uniform per layer. The win is measured as aggregate task quality under a latency target.

Three things are new relative to prior work:

  1. The trigger for changing fidelity is system load, not input difficulty.
  2. The decision uses a curve measured offline per adapter, so no learned router is needed.
  3. Training covers every prefix, so there is no gap between the configurations used in training and at inference.

The idea borrows a lens from classical automated modeling. Approximation Reformulations shows that moving between simpler and richer physical models is well behaved when the simple model is a limit of the rich one along a continuous "fitting parameter." Examples are friction going to zero, or rope elasticity going to infinity. Choosing a model then reduces to moving along that one parameter.

For adapters, the analogous parameter is the scale on each component in the SVD-style form $\Delta W = P\Lambda Q$ used by Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning. Here $P$ and $Q$ hold the component directions and $\Lambda$ is a diagonal of per-component scales. Driving trailing entries of $\Lambda$ to zero is exactly the "simpler model as a limit" structure. Likewise, Automated Model Selection Using Context-Dependent Behaviors makes "simplest adequate model" search tractable by insisting that simpler models be strict subsets of richer ones. Nested prefixes give us that subset property for free. These papers motivate the framing; they supply no neural technique.

Research Program

Training rank-elastic adapters. Each adapter uses the $P\Lambda Q$ parameterization with a maximum rank $R$, for example 64 per matrix, across all $L$ adapted matrices. During training we sample a global component budget $b$ and keep only the top-$b$ components under a global ordering $\pi$ across all layers. We then minimize the task loss of the truncated model:

$$\min_{P,\Lambda,Q}\; \mathbb{E}_{b\sim p(b)}\, \mathcal{L}\big(W_0 + \textstyle\sum_{j\in \pi_{1:b}} \lambda_j\, p_j q_j^\top\big),$$

In this objective, $\lambda_j$, $p_j$ and $q_j$ are component $j$'s scale and directions. $\pi_{1:b}$ is the set of the $b$ highest-ranked components across all layers. $p(b)$ is a sampling distribution over budgets that is weighted toward the full budget, which protects full-rank quality.

The ordering $\pi$ is re-estimated periodically from a stabilized importance score, following the running mean-over-variance signal-to-noise score of IGU-LoRA. We reuse that method; we do not claim it.

This differs from the random batch-level rank sampling of DyLoRA+ in Flexi-LoRA in one respect: truncation is heterogeneous across layers. AdaLoRA found that feed-forward and upper layers need more rank. A global ordering lets the dial drop cheap lower-layer attention components first.

Measuring fidelity curves. After training, we evaluate each adapter at a grid of budgets on held-out data. This gives a quality curve $q_i(b)$ for adapter $i$. The paper's first scientific output is the shape of these curves across tasks. We will report where they saturate and how they compare with the curve from naive post-hoc truncation.

Load-adaptive allocation. At each scheduling interval, the server solves the following problem, where $w_i$ is adapter $i$'s current request share, $c(b_i)$ is the memory or bandwidth cost of serving $b_i$ components, and $C$ is the budget left after the base model and the key-value cache:

$$\max_{\{b_i\}} \sum_i w_i\, q_i(b_i) \quad \text{s.t.} \quad \sum_i c(b_i) \le C.$$

If each $q_i$ is concave, which is the diminishing-returns regime, then greedily adding the component with the highest marginal gain per unit cost is optimal. That is the standard result for separable concave resource allocation. Where a curve is not concave, we use its concave hull.

We implement this on top of S-LoRA. Its unified paging and multi-size batched kernels already handle mixed ranks without padding, so only a prefix of each adapter's pages needs to be resident or read. The contrast is with S-LoRA and Punica. Both treat rank as immutable metadata; we treat it as a schedulable resource.

Workload. We will train a pool of at least 100 adapters on Llama-3.1-8B. The pool covers GSM8K, HumanEval-style code data, the Commonsense170K tasks, and GLUE-style classification tasks, with additional variants trained on data subsets and different seeds. Requests will follow skewed popularity with injected load spikes. The file lists no public multi-adapter pool, so we will release ours.

Baselines

Fixed small rank (naive reference). Every adapter is vanilla LoRA at rank 8, served by unmodified S-LoRA. Its learning rate is swept with the same budget as every other method. This is the "do nothing" point, and it is strong. A Unified Study of LoRA Variants reports that vanilla LoRA on Llama-3.1-8B-Base reaches 75.51% on GSM8K at learning rate 5e-4, and that it has the lowest memory and training time among the variants studied. Our method must beat rank 8 on aggregate quality at equal serving cost to justify itself.

Fixed full rank with paging. Every adapter is vanilla LoRA at rank 64, served by S-LoRA's standard eviction and queueing. This baseline represents the quality ceiling at low load. It pays in queueing delay or dropped requests under spikes. We compare at a fixed p99 per-token latency target and count requests that miss the target as failures.

Post-hoc truncation of vanilla LoRA. A rank-64 vanilla adapter is decomposed with SVD after training, and its trailing components are cut under the same global ordering and the same scheduler. This is the key ablation. It isolates whether elastic training matters, or whether any adapter truncated after the fact already gives a good curve.

Per-budget oracle and closest prior method. The per-budget oracle trains a separate AdaLoRA adapter at each budget level and switches between them. It is an upper bound on quality at every budget, but it costs several times more training and storage. The closest prior method is the DyLoRA+ batch-rank sampling from Flexi-LoRA, which makes the same prefixes valid but truncates uniformly across layers. It isolates our second ingredient, global cross-layer ordering. Flexi-LoRA reports 84.00% average math accuracy on a 3B model using about 31–33% of LoRA-8's parameters. Its full difficulty-aware router is not our target, since it needs difficulty labels.

Quantized adapters. Each rank-64 adapter is quantized to 4 bits, which gives a comparable reduction in memory and bandwidth.

Fairness and the winning condition. All methods use the same data, the same learning-rate sweep, and the same serving hardware and kernels. Beating the baselines means two things. At full budget, elastic adapters must stay within 0.5 points of vanilla rank-64 on each task. At matched memory or bandwidth, they must strictly dominate post-hoc truncation, fixed rank 8 and quantization on the aggregate quality-versus-throughput frontier.

Why not just…?

Why not just truncate an ordinary LoRA adapter with SVD after training? This is the most serious threat. Vanilla LoRA updates may already have fast-decaying spectra. If they do, post-hoc truncation gives curves nearly as good as elastic training, and the training change is unnecessary. The ablation above settles the question directly. If the two curves match within noise, we will drop elastic training and publish the scheduler plus the curve measurements alone. That would still be a useful but smaller contribution.

Why not just use rank 8 everywhere? The LoRA-variants study suggests that small ranks with a tuned learning rate are hard to beat. If no task shows a meaningful gap between rank 8 and rank 64, there is nothing for a dial to trade. There is some evidence that capacity helps on hard tasks. GoRA reports 75.74 on GSM8K at rank 128, against 73.69 for full fine-tuning. We will measure the gap per task before building anything else. A flat result across the pool is a negative finding that ends the project.

Why not just quantize adapters instead? Quantization and truncation reduce the same bytes, and quantization needs no retraining. It is a real competitor. However, it offers one fixed operating point, not a continuum. It is also compatible with truncation, since quantizing a prefix is possible. We compare the two alone and combined on the same frontier. If quantization dominates everywhere, we will say so.

Why not route by input difficulty, as Flexi-LoRA does? Difficulty routing needs task-specific difficulty labels and applies one rank to all layers. Its gains also vary by domain: large on math, smaller on question answering. More fundamentally, it answers a different question, namely which inputs need capacity. Our question is how to share a scarce memory budget across adapters when load rises. The two could be combined later. We keep them separate so that the effect of load-driven allocation can be measured cleanly.

Decisive Checks

The first check comes before any system work. On GSM8K, HumanEval and the commonsense tasks with Llama-3.1-8B, we will test whether quality at rank 64 exceeds quality at rank 8 under tuned learning rates, and whether elastic training beats post-hoc truncation at intermediate budgets. If either answer is no, the project narrows or stops.

The second check measures the cost of elasticity: the full-budget quality penalty of elastic training, which must stay within 0.5 points.

The third check is the end-to-end serving experiment. We replay a skewed, spiky trace over the 100-adapter pool and plot aggregate quality against throughput at a fixed p99 latency target. We then check whether load-adaptive allocation shifts this frontier outward relative to full-rank S-LoRA paging, fixed rank 8, and 4-bit adapters.

We will also report whether the measured curves are concave, since the greedy scheduler's optimality depends on it. Finally, we will report whether the global cross-layer ordering drops lower-layer attention components first, as AdaLoRA's findings predict.