When a medical AI model is unsure, a common safety design is to wrap it in a decision layer that either acts on the prediction or defers the case to a clinician. The layer is calibrated so that the cases it acts on meet a fixed error budget. That guarantee covers only the cases the model handles. Every deferred case lands in a human's queue, and queues accumulate. A backlog that grows faster than it drains produces tired, rushed reviewers, and tired reviewers make more mistakes and follow alerts less often. Their decisions then become the labels used to recalibrate the layer. The safety layer is therefore part of a feedback loop that its own guarantee never sees.
This note proposes a deferral layer whose decisions depend on how much work is already sitting with the reviewers. It borrows the central device of system dynamics, the field that models systems as accumulations ("stocks") changed by rates ("flows") inside feedback loops. The reviewer backlog and reviewer fatigue are modeled as explicit stocks that the layer reads before each act-or-defer decision. The claim is narrow and testable. An error guarantee stated for the model alone overstates the safety of the whole human–AI pipeline under load. A layer that tracks these stocks keeps end-to-end error near budget at equal or higher action volume. This should hold specifically because fatigue has memory, so a memoryless capacity cap should not do as well.
Core Claim
Treat the deferral layer as one tier of a layered system: a model tier below, a human-review tier above, and a narrow interface between them. Standard practice puts only predictions and confidence scores on that interface. We argue the interface must also carry the state of the tier above, meaning how many cases are queued and how degraded the reviewers are. Without that state, the layer's error control fails exactly in the high-load regimes where it matters most.
Write the end-to-end error at time $t$ as
$$E_t = a_t \, \epsilon^{\text{model}}_t + (1-a_t)\, h(F_t),$$
where $a_t$ is the fraction of cases the layer acts on, $\epsilon^{\text{model}}_t$ is the error rate on acted cases, which existing conformal layers control, and $h(F_t)$ is the reviewer error rate as a function of a fatigue stock $F_t$. Fatigue fills with backlog and drains with recovery:
$$F_{t+1} = F_t + \beta B_t - \gamma F_t, \qquad B_{t+1} = B_t + (1-a_t) N_t - c,$$
where $B_t$ is the deferred backlog, $N_t$ is incoming cases, $c$ is reviewer throughput, $\beta$ is how fast backlog turns into fatigue, and $\gamma$ is the recovery rate. Because $F$ has memory when $\gamma < 1$, a burst of deferrals raises reviewer error for many periods after the backlog has cleared.
This produces two loops the static layer ignores. The first is reinforcing: more deferral raises fatigue, fatigue raises reviewer error, and the noisier human labels then make the layer less confident, which leads to more deferral. The second is a balancing loop the layer can exploit: acting on more borderline cases when fatigue is high relieves the stock. The proposed layer chooses $a_t$ to keep the forecast $E_t$ under budget over a short horizon, given the current $B_t$ and $F_t$.
Grounding in Prior Work
The method we build on is STRATCP from Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models. It wraps an EHR foundation model in an action arm and a deferral arm. The action arm uses conformal selection, which picks the cases it acts on while controlling the false-discovery rate. The deferral arm returns calibrated prediction sets. STRATCP assumes exchangeable data and treats the deferral arm as a terminal sink. We keep its action-arm machinery unchanged and add the downstream stock state as an input.
Two papers motivate the gap. Optimizing the Implementation of Clinical Predictive Models to Minimize National Costs: Sepsis Case Study tunes alert thresholds per diagnostic category. It reports that improving physician adherence alone would add about $1 billion in savings, which is direct evidence that the human tier's behavior is a first-order term. It nonetheless treats adherence as a fixed parameter rather than a state driven by alert volume. A Classification of Feedback Loops and Their Relation to Biases in Automated Decision-Making Systems shows formally that model decisions feed back into the data a model later sees. In a synthetic recommender, one such loop collapsed a group's representation from 50% to 8.9%. Its analysis is a taxonomy plus synthetic simulation. Our deferral-to-label loop is a concrete instance of its "outcome" and "sampling" loops, embedded in a working decision layer.
On the architecture side, A Functional Software Reference Architecture for LLM-Integrated Systems places guardrails and monitoring in cross-cutting sidecars, which are separate components that watch requests passing through. Those sidecars observe traffic but do not hold accumulated state about the tiers they protect. Our layer is in effect a guardrail sidecar with stocks.
The closest existing work combining system dynamics with a layered technical system is Security issues of the gold industry chain based on smart blockchain in the context of the Internet of Things. It runs a stock-flow model beside a sensing tier and a blockchain tier, but only as an offline forecaster that never feeds decisions back. We run the stock model online inside the decision path.
Profits Management for Reverse Supply Chain with Supply Disruption constrains the design in a useful way. In multi-tier stock-flow systems, buffered inventory can produce a "false peak" in one tier's metrics after a disruption, and collapse appears only beyond a delay threshold of 5 weeks. The analogue here is that a static layer may look fine on short evaluation windows while backlog builds. Our evaluation must therefore run long enough for delayed effects to appear.
Research Program
We would build three components. The first is a closed-loop replay harness. It streams EHRSHOT patients through CLMBR-T-base and a decision layer. Deferred cases enter a simulated reviewer tier with finite throughput $c$, a fatigue stock, and an error function $h(F)$. The reviewer's labels, not the ground truth, are what get fed back for recalibration.
Because no dataset in the file measures clinician error as a function of load, we will not pretend to know $h$. We sweep it over a family running from flat (no fatigue effect) to steep, with recovery rates $\gamma$ from memoryless ($\gamma=1$) to slow. We report results as a map over this family. The flat-$h$ corner is a built-in control: there the proposal should reduce exactly to STRATCP.
The second component is the stock-aware layer. It is STRATCP's action arm with the selection level adjusted each period by a small model-predictive rule. The rule forecasts $B$ and $F$ a few periods ahead from the equations above and picks the least aggressive action rate whose forecast $E_t$ stays under budget. To estimate $\beta$ and $\gamma$ online, the layer uses only quantities it can observe: queue length, reviewer turnaround, and reviewer–model disagreement on deferred cases.
The third component is an analysis of when the static guarantee breaks. We would derive the backlog level at which $h(F)$ overtakes $\epsilon^{\text{model}}$. Below that level, deferral lowers end-to-end error. Above it, deferral raises it. We would then check that crossover against the simulations.
Baselines
The naive reference point is a uniform-threshold policy that acts on every case above a single fixed score cutoff and defers the rest. This is the analogue of the uniform 80% sensitivity rule that the sepsis study found costs about $700 million more than per-category thresholds (Optimizing the Implementation of Clinical Predictive Models). We run it with its cutoff tuned on the same calibration split. It shows how much any layer helps at all.
The second baseline is marginal conformal prediction, a single global calibration. On EHRSHOT pancreatic cancer at $\alpha=0.10$ it reaches only about 10% selected coverage for the positive class (Act or Defer). We include it to reproduce that known failure inside our harness, which checks that the harness matches published behavior.
The strongest existing method is static STRATCP. On the same benchmark it maintains over 99% selected coverage for pancreatic cancer at $\alpha=0.10$ and acts on nearly 2.5 times more short-length-of-stay patients than standard conformal prediction (Act or Defer). We run it with the authors' stratification, the same CLMBR-T-base embeddings, the same calibration data, and the same number of hyperparameter trials as our layer. We give it periodic recalibration on reviewer-supplied labels at the same cadence our layer uses.
The proposal beats it if two things hold under the moderate and steep fatigue settings. End-to-end error $E$, averaged over the run and at its worst sustained window, must stay at or below 0.10 where STRATCP's exceeds it. Action volume must be no lower than STRATCP's. In the flat-fatigue setting, the two must be statistically indistinguishable.
The key ablation is a memoryless capacity cap. It is STRATCP plus a hard rule that defers at most $c$ cases per period, which corresponds to our layer with $\gamma=1$ and no fatigue stock. This isolates whether accumulated state, rather than mere load awareness, is doing the work. A second ablation feeds the layer the backlog stock but not the fatigue stock.
Why not just…?
Why not just cap deferrals at reviewer capacity? This is the most serious threat, and the ablation above tests it directly. If fatigue in the real tier is memoryless, a cap captures nearly all of the benefit and the system-dynamics component is unnecessary. We expect the cap to lose only when recovery is slow. In that regime a cap allows bursts that keep reviewer error high for many periods afterward, the delayed-damage pattern seen in Profits Management for Reverse Supply Chain. If the cap matches our layer across the slow-recovery settings, we will report that and recommend the cap.
Why not just recalibrate the conformal layer online as new labels arrive? Frequent recalibration does address drift, but it recalibrates on labels produced by fatigued reviewers. The noise it absorbs is the very error it should be guarding against, so the reinforcing loop speeds up rather than breaks. No reported result on Lacuna was found for online-recalibrated STRATCP under label feedback. Our recalibrated-STRATCP baseline is that test. If it holds end-to-end error under budget without stock information, our claim fails.
Why not just tune per-group static thresholds more carefully, as in the sepsis study? Per-category tuning clearly beats a uniform rule (Optimizing the Implementation of Clinical Predictive Models). Any static threshold, however well chosen, still fixes deferral volume in proportion to arrivals. It cannot respond when a surge pushes backlog past the crossover level. The surge experiments below separate the two approaches. Under steady arrivals we expect careful static tuning to be competitive, and we will say so.
Proposed Experiments
All experiments use the EHRSHOT length-of-stay, pancreatic cancer, and thrombocytopenia tasks with CLMBR-T-base. Patients are replayed in their recorded order where it is available and in seeded random order otherwise. Each run spans enough periods for slow fatigue to reach steady state, which we confirm by checking that $F$ has plateaued in the static baseline. Metrics are end-to-end error $E$ (time-averaged and worst sustained window), action volume, peak and mean backlog, and the gap between STRATCP's claimed error and its realized end-to-end error. That last metric quantifies how much the model-only guarantee overstates safety.
The first decisive check is the flat-fatigue control, in which our layer must match STRATCP. The second is a sweep over steepness of $h$ and recovery rate $\gamma$, in which the gain over the memoryless cap should grow as $\gamma$ falls. If it does not, the stock component is unjustified. The third is a surge test: arrivals double for a stretch of periods and then return to baseline. Our prediction is that static STRATCP shows a short-term "false calm" followed by elevated end-to-end error after the surge ends, while the stock-aware layer acts more during the surge and avoids the after-effect. The fourth is a misspecification test: the layer's internal $\beta$ and $\gamma$ are deliberately set wrong by a factor of two, to measure how much of the benefit survives when the stock model is imperfect.
The result we would publish whichever way it comes out is a map of the fatigue-model regimes, showing where load-unaware deferral layers silently miss their error budget and how much of that shortfall a simple cap recovers compared with a stock-aware layer.
References
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- Optimizing the Implementation of Clinical Predictive Models to Minimize National Costs: Sepsis Case Study
- A Classification of Feedback Loops and Their Relation to Biases in Automated Decision-Making Systems
- A Functional Software Reference Architecture for LLM-Integrated Systems
- Security issues of the gold industry chain based on smart blockchain in the context of the Internet of Things
- Profits Management for Reverse Supply Chain with Supply Disruption