Many deployed services run one large model as a shared backbone and attach a small fine-tuned add-on, a LoRA adapter, for each customer task. These are low-rank weight updates layered on top of the frozen model. When more tasks are active than GPU memory can hold, the server must fetch adapters from host memory, and the whole batch stalls. Today's best fix is admission control: refuse to start requests whose adapter is not resident. That protects priority traffic but leaves everyone else waiting for seconds. We propose a third option borrowed from neuroscience. Give every tenant a tiny linear readout, a single matrix of a few kilobytes, attached to an intermediate layer of the shared backbone, and keep all of these readouts permanently in memory. When the server is overloaded, requests whose adapter is not resident are answered by their readout, exiting the network early, instead of waiting. A feedback controller watches queue latency and decides how much traffic takes this cheaper path.
The biological motivation is specific. Untrained macaques have never seen text, yet a simple linear readout from their inferior temporal cortex already separates English words from pseudowords at 76% accuracy. Trained baboons reach 72%, and the same readout from earlier area V4 gets only about 57% (Kar et al.). In networks, features learned for general object recognition explain human letter perception better than features trained on letters. The best fit comes from mid-to-late layers, and adding letter-specific fine-tuning or branches does not help (Janini et al.). Brains handle new tasks by reading out existing intermediate representations, not by loading new circuitry. Our claim is that the same move gives multi-tenant serving a graceful-degradation tier that costs no extra memory traffic. For a large class of tenant tasks, it recovers most of the accuracy their adapters provide.
Core Claim
For classification-style tenant tasks served on a shared backbone, a frozen linear readout at a well-chosen intermediate layer should recover a large fraction of each adapter's accuracy over the base model. Such tasks include intent detection, moderation, attribute tagging and scoring, which return a label rather than free text. A serving engine that routes overflow traffic to these readouts, under a queue-latency controller, should then deliver more correct answers within latency targets than either admission gating or adapter compression. We make no claim for open-ended generation. A linear readout cannot write paragraphs, and those tenants stay on the existing gating path.
The new element is not early exit, slimming, or LoRA batching, each of which exists. It is the use of intermediate-layer linear decodability as a separate, always-resident tier for different tasks, invoked by a controller precisely when adapter residency fails.
Relation to Prior Work
The serving systems supply the problem and the strongest competitor. Punica batches requests for many different adapters in one GPU step and reports 12x the throughput of HuggingFace PEFT. It assumes every adapter is resident, so it does not address what happens when they are not. CLIMB diagnoses the "residency cliff": when the number of active adapters exceeds the number that fit on the GPU, on-demand fetching stalls the batch. CLIMB fixes this with feasibility-first admission gating. By its own account, that shifts seconds-long delays onto background tenants. We keep CLIMB's gating for priority and generative traffic and add a path for the requests it would otherwise park.
Algorithmic elasticity supplies the control idea. The pub/sub work of Cao et al. skips rare predicates during traffic spikes under a PID-style controller. It halves latency (57.11 ms to 27.32 ms) while keeping false positives below 0.1%. Hardware scaling is too slow for such bursts, a point also made by SNF for network functions. We transplant the "trade bounded precision for speed under queue pressure" controller from boolean matching to neural readouts.
Elastic networks constrain the design. Slimmable and universally slimmable networks, NestDNN, and LayerDrop shrink one task's model on demand, and all require special training of the backbone. LayerSkip and EESD reuse early layers only to draft tokens for the same generation task. Our readouts need no backbone retraining and serve different tasks.
FairGrad treats multi-task optimization as fair allocation of a shared resource, but only offline during training. We borrow its framing only for evaluation, reporting per-tenant worst-case accuracy loss alongside the average.
Research Program
The first component is a measurement study of how much accuracy the reuse tier can recover. For each tenant task $\tau$ and layer $l$, we fit a linear readout on frozen layer-$l$ features. We then define recovery as the fraction of the adapter's gain the readout retains:
$$R_\tau(l) = \frac{a_{\text{readout}}(\tau,l) - a_{\text{base}}(\tau)}{a_{\text{adapter}}(\tau) - a_{\text{base}}(\tau)}.$$
Here $a$ is held-out accuracy, and $a_{\text{base}}$ is the backbone answering with a task prompt and no adapter. The neuroscience predicts that $R_\tau(l)$ peaks at mid-to-late layers rather than at the output, as in Janini et al. and the V4-versus-IT contrast. A peak well before the final layer also matters for systems: it means the reuse tier saves compute as well as memory traffic.
The second component is the engine. It extends a CLIMB-style ingress so that each arriving request is assigned to one of three paths. A request goes to its resident adapter if one is available, or waits behind the gate if it is a priority or generative request. Otherwise it is sent through the backbone to its readout layer and answered there. Readout requests at layer $l$ can be batched with ordinary traffic up to that layer, and their exit frees the remaining compute.
The third component is the controller. It sets the fraction $r(t)$ of eligible overflow traffic diverted to readouts, using a PID update on the gap between observed p99 queue delay $q(t)$ and the latency target $q^*$:
$$r(t) = \mathrm{clip}_{[0,1]}\Big(k_p e(t) + k_i \textstyle\sum_{s\le t} e(s) + k_d (e(t)-e(t-1))\Big), \quad e(t) = q(t) - q^*.$$
A per-tenant guard excludes any tenant whose validated $R_\tau$ falls below a contracted floor. This turns "degrade gracefully" into an explicit accuracy bound, analogous to Cao et al.'s bounded false-positive rate.
Baselines
The naive reference is global FIFO continuous batching with on-demand adapter loading. On CLIMB's stress test this is the cliff itself: an RTX 5090 with eight active adapters and room for four, running Qwen2.5-7B with rank-128 adapters, where engine latency reaches 38.4 s. Every system is replayed on identical arrival traces, hardware and adapters.
The strongest existing method is CLIMB. On the setup above, CLIMB reports priority-tenant TTFT p99 falling from 38.7 s to 13.1 s, engine latency of 0.13 s, and throughput of about 10.66 requests per second (CLIMB). We run CLIMB's admission policy unchanged inside our engine, so the only difference is the readout path. Beating it means three things together. Priority TTFT p99 must be no worse than CLIMB's on our replay. Background-tenant p99 latency must drop substantially. Accurate goodput, defined as correct answers per second returned within the latency target, must rise. The accuracy loss on diverted requests must stay within each tenant's floor.
A second strong competitor is rank truncation. Each tenant's rank-128 adapter is compressed by SVD to a small rank so that all of them fit in memory, and these compressed adapters serve overflow traffic. It gets the same memory budget as our readouts and the same controller, so the comparison isolates "readout from an intermediate layer" against "small adapter through the full network."
A third baseline is prompt-only fallback: overflow requests go to the bare backbone with a task instruction. It costs nothing to store and shows whether the readout adds anything over the base model.
Two ablations isolate the key ingredients. The first places the readout at the final layer instead of the layer chosen by $R_\tau(l)$, testing whether intermediate-layer reuse matters beyond memory savings. The second replaces the PID controller with a static utilization threshold, which Yin et al. argue causes oscillation.
Why not just…?
Why not just keep heavily truncated LoRA adapters resident for every tenant? This is the most serious threat. It preserves generative ability, which readouts cannot. A truncated adapter still runs the full network depth and goes through adapter batching kernels, so its compute cost under overload is that of a normal request. A readout at layer $l$ exits early. The experiment decides this directly. If truncated adapters match readouts on accurate goodput for classification tenants, the claim fails for that class. The defensible contribution would then shrink to a combined ladder that uses readouts only where $R_\tau(l)$ is high and the exit is early, and we would report it that way.
Why not just answer overflow requests with the base model and a good prompt? It is free, and for some tasks it may be enough. But Kar et al.'s V4-versus-IT contrast and Janini et al.'s layer profile suggest that task-relevant information is often linearly available at intermediate layers before it shows up as correct prompted output. The prompt-only baseline and the recovery ratio, whose denominator is defined against it, show per task whether the readout earns its kilobytes.
Why not just use CLIMB with more resident slots, better prefetching, or more GPUs? More slots only move the cliff; the working set of adapters can always exceed them. Adding machines takes tens of seconds, while bursts last seconds (Cao et al.; SNF). We sweep the number of resident slots, keeping the number of active adapters above it, to show where, if anywhere, gating alone catches up.
Why not just use early exit as in LayerSkip? LayerSkip exits early for the same task and requires training the backbone for it. Our readouts attach to a frozen backbone and serve different tasks, so there is no retraining and no interference with other tenants. The final-layer ablation separates the benefit of early exit from the benefit of reuse.
Proposed Experiments
The first experiment is a vision reuse study that directly tests the transfer of the neuroscience result, using datasets in the file. On an ImageNet-trained backbone, we compute $R_\tau(l)$ for letter classification on GoogleFonts and EMNIST, replicating Janini et al.'s layer profile as an accuracy measure. We repeat this for CelebA's 40 binary attribute tasks, treated as 40 tenants, each with a LoRA adapter as the upper reference. FairGrad's reported CelebA result (mean rank 4.95, 0.37% average drop) is a jointly trained multi-task reference, not a serving baseline. The main measure is the distribution of $R_\tau$ across the 40 tenants and the depth at which it peaks.
The second experiment uses the language backbone. We reproduce CLIMB's stress test on Qwen2.5-7B with rank-128 adapters, mixing classification-style tenants with generative tenants. Arrival lengths and timing are driven by the ShareGPT trace used by Punica and MUXWISE, with bursts injected at the 60% minute-to-minute and 767% spike magnitudes reported by Cao et al. The file contains no text-classification tenant benchmark, and no reported result found on Lacuna for readout accuracy on Qwen2.5-7B intermediate layers. Tenant tasks must therefore be constructed and released with the code.
The decisive checks follow from the claim. The idea fails if median $R_\tau$ at the best layer is low, or if that layer is near the output so no compute is saved. It also fails if rank-truncated adapters match readouts on accurate goodput at equal memory. It succeeds if, at CLIMB-matched priority TTFT p99, background p99 latency drops and accurate goodput rises with every diverted tenant above its accuracy floor.
References
- The inferior temporal cortex is a potential cortical precursor of orthographic processing in untrained monkeys
- General object-based features account for letter perception
- Punica: Multi-Tenant LoRA Serving
- CLIMB: Taming the LoRA Residency Cliff in Multi-LoRA Serving
- Adjusting Matching Algorithm to Adapt to Dynamic Subscriptions in Content-Based Publish/Subscribe Systems
- SNF: serverless network functions
- Slimmable Neural Networks
- Universally Slimmable Networks and Improved Training Techniques
- NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision
- Reducing Transformer Depth on Demand with Structured Dropout
- LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding
- Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism
- Fair Resource Allocation in Multi-Task Learning
- An Adaptive Elastic Multi-model Big Data Analysis and Information Extraction System
- Towards High-Goodput LLM Serving with Prefill-decode Multiplexing