Who Gets Processed First? Scheduling-Induced Bias in Brain–Behavior Studies

Large neuroimaging studies run every scan through a long preprocessing pipeline before anyone can ask how brain measurements relate to clinical scores. Compute is finite, and the cluster scheduler (the software that decides which waiting job runs next) usually tries to finish as many scans per hour as possible. The cheapest way to do that is to run the fastest jobs first. Scans that take longest to process are often the ones with tumors, strokes, heavy atrophy, or thick-slice clinical acquisitions, and those come from the sickest or oldest patients. If they systematically finish last, any analysis run before the queue drains is computed on a cohort skewed toward easy anatomy. That includes interim results, deadline-driven analyses, and studies whose compute budget runs out. The estimated link between brain structure and symptoms is then biased in a direction nobody chose.

We propose to measure this effect and remove it at the scheduler. The scheduler runs predicted-short scans first for throughput. It also reserves a small, fixed share of capacity to process the predicted-long scans in a pre-committed random order. Because the order is random and known, every scan's chance of having been processed by any time is known. A standard survey-sampling estimator can then give unbiased, honestly uncertain brain–behavior correlations at any moment while the queue is still draining. The intended result is a scheduler whose processing order does not distort the science, at a small, measured throughput cost.

Core Claim

Scheduling research treats fairness as bounded slowdown. Stretch, defined as response time divided by service time, captures this. Long jobs must not starve, and Competitive Fair Scheduling with Predictions shows that worst-case stretch and good average response can coexist even with noisy runtime predictions. Clinical neuroimaging treats fairness as equal diagnostic error across patient groups, as in Fair CCA for Fair Representation Learning: An ADNI Study and How fair is your graph?.

Our claim joins the two. In a clinical cohort, job size is plausibly correlated with clinical severity and demographics. If so, a size-based scheduler is also a selection mechanism. Bounding stretch is then not enough, because a deterministic short-first order still yields a biased interim sample, even if every job eventually runs. What is needed is a probabilistic guarantee: every subject must have a known, nonzero probability of having been processed by time $t$.

The mechanism adapts one idea from the $RG^+$ algorithm in Competitive Fair Scheduling with Predictions. That algorithm runs Round Robin on a fraction $\epsilon$ of processor speed as a safety net, which bounds the worst-case competitive ratio. We reuse the reserved $\epsilon$ slice but change its purpose. In $RG^+$ it gives a competitive-ratio guarantee. Here it gives a statistical one, because the slice processes a uniformly random sample of the slow tail.

Research Program

The work has three parts.

The first part is an empirical audit. It tests whether the problem exists, and the proposal stands or falls on it. We will run DeepPrep and "Recon-all-clinical" on ADNI and on the clinical pathology cohorts described in those papers (glioma, stroke, disorders of consciousness, thick-slice multi-contrast scans). We will log per-stage wall time and failures, including retries. We then measure how runtime relates to diagnosis (cognitively normal, mild cognitive impairment, Alzheimer's disease), to cortical thickness, to tau burden, to age, and to sex.

Two features of the pipelines make a dependence plausible. Recon-all-clinical's deep network is fast, but it is followed by classical meshing and topology correction, and the cost of that step plausibly grows with anatomical abnormality. DeepPrep reports that clinical cases with severe distortion completed at only 69.8% under fMRIPrep, which shows that difficulty concentrates in pathological scans. If runtime turns out to be nearly independent of clinical variables, scheduling order is harmless and the proposal should stop there.

The second part is a runtime predictor built only from information available before processing. Its inputs are scan metadata (contrast, slice thickness, voxel size, site) and outputs of a seconds-long first pass such as SynthSeg volumes, a lesion-presence flag, and ventricle size. The predictor follows the gated short/medium/long design and conformal calibration used in ATLAS. Its outputs serve two purposes: they drive the throughput part of the scheduler, and they define the strata used by the estimator.

The third part is the scheduler and estimator. Before the run starts, subjects are split into strata by predicted runtime. On $1-\epsilon$ of capacity, the scheduler processes the short strata in shortest-predicted-first order. On the reserved $\epsilon$, it works through each long stratum in a random order drawn in advance. At any time $t$, the processed members of a long stratum are a simple random sample of that stratum, so their inclusion probability is known. The statistic of interest is a population quantity, for example the covariance between regional cortical thickness and a clinical score. It is estimated as

$$\hat{\theta}(t) = \sum_{h} \frac{N_h}{n_h(t)} \sum_{i \in S_h(t)} g(x_i, y_i),$$

where $h$ indexes strata, $N_h$ is the number of subjects in stratum $h$, $S_h(t)$ is the set processed by time $t$ with $n_h(t)$ members, and $g$ is the per-subject contribution: an imaging feature $x_i$ paired with a clinical score $y_i$. Correlations, regression slopes and diagnostic AUC are ratios or smooth functions of such totals. Their variance comes from standard stratified-sampling formulas, so confidence intervals are honest at every interim time.

The reason for stratifying on predicted runtime is that strata must be fixed before processing. If they depended on realized runtime, the inclusion probabilities would no longer be known. Prediction error therefore costs precision but not bias, which separates this design from the error-sensitive bounds in the scheduling literature. Those bounds depend on prediction error as $O(\eta^3\sqrt{P})$, where $\eta$ is prediction error and $P$ is the ratio of the largest to smallest job size, and ATLAS reports that max-stretch is hypersensitive to single mispredictions.

The share $\epsilon$ sets a three-way tradeoff between subjects per hour, worst-case stretch of long jobs, and estimator variance. We will map this tradeoff explicitly rather than tune it once.

Baselines

The naive reference point is first-come-first-served dispatch in submission order, which is what Nextflow does in DeepPrep. If submission order is unrelated to clinical status, the interim sample is roughly random and unbiased, but throughput is poor. ATLAS reports that a non-clairvoyant Round Robin reaches a total completion time ratio of 1.975, where 1.0 is the optimum. We include both first-come-first-served and a shuffled-order variant. The shuffled variant is the cleanest unbiased comparator, because it is simply a random sample processed at baseline speed.

The strongest throughput competitor is shortest-predicted-job-first using an ATLAS-style predictor. On the Alibaba PAI-2020 trace, ATLAS reports a total completion time ratio of 1.106 for this strategy, and an 11x max-stretch improvement for its Meta-SPRPT variant. These are the throughput and stretch numbers our scheduler must approach. The strongest stretch-fairness competitor is $RG^+$ from Competitive Fair Scheduling with Predictions. That paper reports that $RG^+$ outperformed Round Robin and Greedy-with-Rounding on cloud traces, but we found no reported numerical result on Lacuna for those runs.

On the question that matters here, interim bias in a clinical correlation, no reported result found on Lacuna for any scheduler. That benchmark has to be built.

All schedulers will receive the same recorded per-scan runtimes, the same predictor, and the same simulated cluster. Background load will be replayed from the Alibaba PAI-2020 arrival pattern to mimic multi-tenant contention. Every method gets the same tuning budget for its one or two parameters on a held-out ADNI split.

Two ablations isolate the key ingredient. The first gives the reserved $\epsilon$ slice to round-robin over long jobs, as $RG^+$ does, instead of random order. This keeps the throughput profile but loses the known inclusion probabilities. The second applies shortest-predicted-job-first followed by post-hoc covariate weighting, so that design-based randomization is replaced by modeling.

Beating the baselines means two things at once. Throughput must be within roughly 10% of shortest-predicted-job-first. At every interim time from 20% to 100% of completion, bias must be indistinguishable from the shuffled-order baseline, and nominal 95% intervals must actually cover the full-cohort value about 95% of the time.

Why not just…?

Why not just process in random order? Random order is unbiased by construction, and it is a genuine threat. If runtimes in a cohort are not very right-skewed, the extra subjects bought by running short jobs first are few. Random order then wins on simplicity, and possibly on variance as well, because the stratified design samples the long tail thinly. Our experiments report precision per compute-hour against shuffled order at matched wall time. We expect a gain only where the runtime distribution has a long tail, as clinical pathology cohorts should, and we will report that regime boundary.

Why not run shortest-first and reweight afterward by age, sex and diagnosis? Early in a shortest-first run, subjects predicted to be long have essentially zero chance of having been processed. No reweighting can recover a group with zero coverage. Any correction must extrapolate a model fitted on easy scans to hard ones, and the hard scans are exactly where the brain–symptom relationship may differ. The fairness literature in this file also warns that models trained on the observed group alone can miss sex and diagnostic disparities: How fair is your graph? found that ignoring sex left true-positive-rate gaps intact. Our second ablation measures how far the reweighting gets in practice.

Why not use a stretch-bounded scheduler like $RG^+$, since it already guarantees long jobs are not starved? Bounded stretch fixes when a job finishes relative to its size. It does not make the set of finished jobs at time $t$ a probability sample. A deterministic order that is fair in the queueing sense can still be systematically unrepresentative clinically. The first ablation tests exactly this.

Why not make preprocessing fast enough that ordering doesn't matter? DeepPrep processed 54,515 UK Biobank scans in 6.5 days on a cluster and runs 11x faster than fMRIPrep. For a single clean cohort on dedicated hardware, the problem may indeed shrink. It does not vanish, for three reasons. Clinical scans still route through slower classical stages and through failure-and-retry loops. Clusters are shared. And Pipeline-Invariant Representation Learning for Neuroimaging shows that results shift by up to 9.2% across pipelines, which pushes studies to reprocess cohorts several times. The audit will show how large the runtime spread is under DeepPrep itself.

Proposed Experiments

The decisive first check is the audit. It asks whether processing runtime, under DeepPrep and Recon-all-clinical, correlates with diagnosis, cortical thickness, tau burden or age on ADNI and on the clinical pathology cohorts. We also check whether shortest-predicted-first, stopped at 25%, 50% and 75% completion, shifts the cortical-thickness-versus-diagnosis effect size or the thickness–tau canonical correlation away from its full-cohort value. As a downstream check, we take the 0.773 ROC-AUC that FR-CCA reports on ADNI and ask whether interim cohorts change that AUC or the demographic parity gap it reduces.

If the audit shows no meaningful shift, we report a useful negative result and stop. If a shift appears, the main experiment replays recorded runtimes through every scheduler under contention and compares five quantities: subjects per hour, max-stretch, interim bias, confidence-interval coverage, and sex and age subgroup coverage over time.

A secondary replication uses ABIDE, which has strong sex imbalance across sites, and ADHD-200 Peking. These test whether the same distortion appears in functional-connectivity analyses, using the 76.15% classification accuracy reported for multiscale markers in Individual classification of ADHD patients as the full-cohort reference.

Finally, we sweep $\epsilon$ and deliberately degrade the runtime predictor. This confirms the central design property: bias stays near zero as prediction error grows, and only variance and throughput degrade.