When Fast Schedulers Become Unfair to Patients: Load-Dependent Demographic Latency in Clinical LLM Serving

Hospitals and health-tech companies increasingly run clinical questions through shared large language model (LLM) serving engines. To keep throughput high, these engines reorder work. They favor short requests, group requests of similar length, and reuse cached computation for prompts that begin with text they have already seen. Each trick is reasonable on its own. This note asks whether, together, they quietly penalize some patients: whether requests describing patients from particular demographic groups wait longer, and whether that gap appears only when the system is busy. There are two plausible routes. Language models can respond at different lengths when a vignette names a patient's gender or ethnicity. Rarer phrasings of a vignette can also benefit less from the cache. A scheduler that never sees demographics could still turn these small, group-linked differences into systematic delay.

The measurement idea comes from neuroimaging. Psychophysiological interaction (PPI) analysis asks whether the coupling between two brain signals changes with the experimental context, and it tests this with an interaction term in a regression. Here we ask the same kind of question: does the link between system load and a request's latency depend on the demographic group the request describes? The test is again an interaction coefficient. A static gap between groups that stays flat with load is mostly a property of the model's outputs. A gap that grows with load is created by the scheduler, and a scheduler can fix it.

Core Claim

The claim is that throughput-oriented LLM schedulers can produce latency disparities across the demographic groups named in clinical prompts, and that these disparities grow with contention. We want a measurement that separates disparity the scheduler creates from disparity it merely passes along. We also claim that schedulers built to bound worst-case relative slowdown can close most of the scheduler-created part without seeing any demographic label, at a small, measurable throughput cost.

The ingredients come from three places. The motivating evidence comes from DiversityMedQA, which adds gender and ethnicity descriptors to medical board questions and finds accuracy drops in several LLMs, pronounced for Llama3-8B on ethnicity. That shows the descriptors change model behavior; nobody has yet measured whether they change latency under a scheduler. The systems setting comes from EWSJF, which sends short queries ahead of long ones for throughput, and from SGLang, whose cache stores prompt prefixes in a tree and evicts the least recently used. The research file names the risk that prefix-favoring policies starve prompts with unfamiliar prefixes as an open problem, but no one has measured it. The remedies come from the fairness-throughput scheduling literature: Competitive Fair Scheduling with Predictions and Revisiting Size-Based Scheduling with Estimated Job Sizes.

Research Program

The central estimand is written like a PPI model. For request $i$ served under scheduler $s$, let $L_i$ be its log end-to-end latency. Let $\lambda_i$ be the arrival rate over a trailing window before $i$ arrived, centered. Let $g_i$ be an indicator for the demographic variant named in the prompt, and let $c_i$ be controls for input length and realized output length. We fit

$$L_i = \beta_1 \lambda_i + \beta_2 g_i + \beta_3 (\lambda_i \times g_i) + \gamma^\top c_i + \epsilon_i .$$

Here $\beta_2$ is the static group gap and $\beta_3$ is the load-dependent gap, which is our measure of scheduler-induced starvation. $\epsilon_i$ is the residual. We fit the model twice. Without the length controls, $\beta_3$ measures total harm. With them, whatever $\beta_3$ remains points to a mechanism other than length, most likely cache reuse.

Two lessons from the PPI literature shape the design. First, Di, Zhang and Biswal show that simple correlation measures of brain coupling mostly reproduce baseline structure; beta-series correlation matches resting-state connectivity at r = 0.91–0.92. By analogy, a raw group-versus-latency correlation will mostly reflect baseline length differences. This is why we estimate a contrast across load levels and do not rely on raw correlation. Second, PPI's known multicollinearity problem, and its practice of forming the interaction on a deconvolved signal, carry over directly. Queueing delay responds to arrival bursts with a lag, so we center $\lambda$, use a trailing window, and check the result across window lengths.

We will build three things. The first is a clinical serving trace. DiversityMedQA vignettes are mixed into the bimodal workload of the EWSJF benchmark, which runs from 32 to 4,096 tokens. Demographic variants appear at set population shares, with one group rare at about 10%. Arrival rates are swept from light to saturated load.

The second is a harness that runs one open-weight model, Llama3-8B, chosen because DiversityMedQA found strong bias in it. The model is served under several scheduling policies on identical hardware.

The third is a decomposition experiment. We move the demographic descriptor from early in the vignette to the end. If the gap remaining after length controls comes from the cache, this move should remove it, because variants that differ only at the end share almost the whole cached prefix.

Baselines

The naive reference is first-come-first-served (FCFS) on vLLM. FCFS cannot starve by group, because it ignores request content, so it gives the floor for scheduler-induced $\beta_3$. It also gives the throughput cost of doing nothing clever: EWSJF reports 30–54% higher token throughput and up to 4x lower average time-to-first-token than FCFS under high contention. The honest question for any fix is how much of that gain it keeps.

The strongest throughput competitors are EWSJF and SGLang's prefix-cache scheduling. SGLang reports up to 6.4x throughput over vLLM on agentic workloads and cache hits at 96% of the theoretical optimum (SGLang). These are the policies most likely to create disparity, because they are the ones that exploit request size and prefix popularity. The file does not say whether EWSJF's code is released, so we will reimplement it from its paper and confirm that it approximately reproduces its reported gain over FCFS on the synthetic benchmark before using it.

The fairness-oriented competitors are two schedulers that never see group labels. The first is $RG^+$ from Competitive Fair Scheduling with Predictions. It needs only a short/long prediction of each request's size and bounds max-stretch, the worst ratio of a request's time in the system to its service time. That paper reports it beating Round Robin on cloud traces, but the file gives no numerical values. The second is FSPE+PS from Dell'Amico et al.. It moves requests that outrun their size estimate into a processor-sharing pool. We include it because under-estimated long generations are exactly the failure we expect for verbose demographic variants.

The key ablation is an oracle, group-aware minimax scheduler. It sees the demographic label and minimizes the worst group's 99th-percentile latency, in the spirit of Minimax Demographic Group Fairness in Federated Learning but applied to queue priority. The gap between it and the group-blind schedulers shows how much blindness costs.

All policies share the same model, GPU, trace seeds, and output-length predictor. Each gets the same tuning budget, matched to EWSJF's reported meta-optimizer convergence of 5–8 trials.

We count as a win a group-blind policy that cuts the $\beta_3$ it would otherwise inherit from EWSJF or SGLang by at least half, is statistically indistinguishable from FCFS on $\beta_3$, and keeps at least half of EWSJF's throughput advantage over FCFS. No reported result found on Lacuna for demographic latency disparity under any LLM scheduler, so the scheduler-induced disparity is itself the first number this work would establish.

Why not just…?

Why not just compare mean latency per group? Averages over a run mix light and heavy load. A gap that exists only under contention gets diluted, and a gap that exists even when the system is idle gets wrongly blamed on the scheduler. The interaction term separates these two cases. If $\beta_3$ is near zero for every policy while $\beta_2$ is not, the honest conclusion is that the disparity comes from the model's output lengths and not from scheduling. We would report that result, and it would redirect effort toward the model.

Why not just use FCFS or processor sharing and accept the throughput loss? Size-blind policies may well remove scheduler-induced disparity. That is why FCFS is the reference point. But EWSJF's reported 30–54% throughput gain means clinical deployments will not give up size-aware scheduling on principle. The useful question is the shape of the tradeoff: how much disparity each unit of throughput buys. We will trace that curve across all the policies.

Why not just give the scheduler the demographic label and enforce quotas? Routing clinical requests by patient ethnicity raises legal and privacy problems, and many serving layers will not have the label at all. The oracle group-aware scheduler is included precisely to measure what blindness costs. If it beats $RG^+$ and FSPE+PS by a wide margin, the case for group-blind stretch bounds weakens. We would state that plainly, not argue around it.

Why not just normalize by length, since stretch already accounts for size? Stretch fairness still lets long requests wait longer in absolute seconds, and absolute delay is what harms a patient. More importantly, a gap that survives length controls cannot be fixed by any length normalization. The descriptor-position experiment tests directly whether such a residual exists and whether it comes from the cache.

Proposed Experiments

The first check establishes the phenomenon. Under EWSJF and SGLang scheduling, is $\beta_3$ positive for the rare demographic variants, and does it vanish under FCFS? We report per-group time-to-first-token, 99th-percentile end-to-end latency, and stretch, with bootstrap intervals over trace seeds.

The second check decomposes the phenomenon. We compare $\beta_3$ with and without length controls, then repeat with the descriptor moved to the end of the vignette. A drop in the residual after the move implicates prefix caching.

The third check measures clinical harm. Under a fixed latency deadline, dropped requests become unanswered diagnoses, so we report the effective accuracy for each group, combining DiversityMedQA's model-accuracy disparity with scheduling losses.

The fourth check tests remedies. $RG^+$ and FSPE+PS are placed on the throughput-versus-$\beta_3$ plane against EWSJF, SGLang, FCFS, and the group-aware oracle.

The result that would settle the claim is a positive, load-dependent $\beta_3$ under throughput schedulers that a label-blind, stretch-bounded policy removes while keeping most of the throughput. A flat $\beta_3$ everywhere would falsify the claim.