Many real decision pipelines look like an editorial board. A cheap automatic screener sees every incoming item. Some items get passed to a handling editor, and a few get passed further to a senior arbiter. Each tier has a fixed amount of expert attention, and items arrive one at a time. A decision to escalate or settle an item is made once and cannot be undone, and nobody knows in advance how many hard cases are still coming. If the board spends its senior attention on early borderline cases, it has nothing left for the truly contested ones that arrive later. If it spends too cautiously, expert time goes unused. This proposal treats the board's escalation rule as a streaming algorithm: a procedure that sees each item once, keeps only a small summary in memory, and must commit as it goes.
The core idea fits in two sentences. Prior work shows that, offline, the best rule for handing an item from a machine to a human is a threshold on how much the human is expected to reduce error compared with the machine. We extend this to a chain of tiers seen in a single pass. Each tier keeps a running threshold on "expected error reduction from escalating", and a pacing signal adjusts that threshold to keep the tier's budget on schedule. We also estimate each tier's error rate only from the items actually sent to it, which raises an exploration problem that offline methods never face.
Core Claim
Differentiable Learning Under Triage supplies the method we build on. It shows that for a fixed classifier, the optimal deferral policy under a human budget $b$ is a deterministic threshold on the gap between model loss and human loss. It also shows that confidence-based deferral wastes budget on items where the human is no better than the machine. Its algorithm, however, alternates gradient steps and threshold updates over many passes of a static dataset. It treats the human side as one homogeneous pool. And it assumes the human's loss is known on every training item.
We claim that a board of $J$ tiers, seen once in arrival order, can get within a small margin of the hindsight-optimal multi-tier triage using three things. The first is a per-tier rule built on estimated loss gaps rather than confidence. The second is an online budget-pacing correction to each threshold. The third is a small amount of forced exploration, which keeps the loss estimates of upper tiers unbiased even though those tiers only ever see what lower tiers send them.
Concretely, an item $x$ sitting at tier $j$ is escalated to tier $j+1$ when
$$\hat g_j(x) \;=\; \hat\ell_j(x) - \hat\ell_{j+1}(x) \;>\; \tau_j(t),$$
where $\hat\ell_j(x)$ is the estimated error of tier $j$ on $x$, $\hat g_j(x)$ is the estimated gain from escalating, and $\tau_j(t)$ is tier $j$'s threshold at stream position $t$. After every arrival, the threshold is nudged so that spending tracks the budget rate:
$$\tau_j(t+1) \;=\; \tau_j(t) + \eta\,\big(s_j(t) - \rho_j\, t\big),$$
where $s_j(t)$ is the number of items escalated to tier $j+1$ so far, $\rho_j$ is that tier's allowed escalation rate, and $\eta$ is a step size. This is a dual-variable (price) update of the kind used for online budgeted allocation. The reason to use rate budgets is that the stream length is unknown, which Optimal Streaming Algorithms for Submodular Maximization with Cardinality Constraints identifies as the central difficulty for one-pass threshold rules.
The threshold is initialized from a small prefix sample, using the sample-then-threshold logic of Prof-K: Probabilistic One-Pass Filtering for Efficient Top-k Selection. That paper sets a conservative cut from a sample whose size scales as $(kN)^{1/3}$. We use the same idea to place the starting threshold near the $(1-\rho_j)$ quantile of $\hat g_j$.
The multi-tier structure matters because the tiers interact. Tier 1's threshold sets the population that tier 2 sees. Tier 2's error estimates are therefore biased toward hard items unless a small random fraction of items is escalated regardless of score.
Research Program
We will build the escalation chain in three tiers and study it in three steps.
The first step is a single-tier analysis under random-order arrival. We will check whether pacing plus prefix initialization achieves vanishing regret against the offline optimal threshold from Okati et al. as the stream grows. This mirrors the random-order recalibration result of Norouzi-Fard et al..
The second step adds tiers and partial feedback. Each tier's loss model $\hat\ell_{j}$ is updated online only from items that tier actually handled, plus an $\epsilon$-fraction of forced escalations. We will measure how the estimation bias in upper tiers depends on $\epsilon$.
The third step stresses the system with non-random orders, which Learning and modeling big data argues are the norm for real streams. We use topic-clustered order, bursts of ambiguous items, and a late surge after early budget exhaustion. In these orders a fixed threshold should fail and the pacing term should matter.
Memory is held fixed throughout. Each tier keeps only its threshold, a bounded loss model, and a running count. Nothing is revisited.
The automatic tier is a fine-tuned text or image classifier. Human tiers are simulated from the multi-annotator records already present in each dataset, as described below, so experts are never spent on simulation. We deliberately do not use LLM agents as tiers in the core study. Policy-as-Prompt shows that GPT-4o-mini moderators with semantically identical prompts disagree on edge cases while keeping aggregate accuracy near 0.79. A tier with that kind of instability would confound the escalation question. We include it only as a secondary condition, described under Proposed Experiments.
Baselines
The naive reference point is no escalation: the automatic tier decides everything. This gives the error floor that any spending of expert budget must improve on.
The second reference is confidence-based escalation with a fixed cutoff, calibrated on the same prefix. It escalates the least-confident items until each tier's budget runs out. This is the "confidence-based triage" baseline that Okati et al. compare against, and it is what most deployed systems would do.
The strongest existing method is offline Differentiable Triage. It is trained on a held-out historical split with full knowledge of human labels and then deployed with its learned threshold frozen. For more than one tier, it is applied tier by tier using each tier's budget. On the Hatespeech and Galaxy Zoo benchmarks the file reports only that it achieved lower misclassification errors than confidence-based and score-based triage across all budgets; no reported result found on Lacuna for its per-budget error values. We will therefore re-run it ourselves on identical splits. It gets the same backbone classifier and the same hyperparameter search budget as our method, measured in tuning runs and GPU hours. It is also given an advantage our method lacks: full offline access to the history.
A fourth reference is the hindsight oracle. It chooses per-tier thresholds after seeing the entire stream with true loss gaps. We use it only to compute regret.
The ablations isolate each ingredient, and all of them keep identical budgets and memory. Replacing $\hat g_j$ with model confidence isolates the loss-gap signal. Setting $\eta=0$ isolates pacing. Setting $\epsilon=0$ isolates exploration. Merging tiers 2 and 3 into one pool holding their combined budget isolates the hierarchy.
Beating the strongest baseline means two things on at least two of the three datasets. First, lower final system error than frozen Differentiable Triage at matched budget, under non-random orders. Second, statistically indistinguishable error under random order. In both cases the win must not come from simply spending more budget.
Why not just…?
Why not just use a fixed confidence cutoff calibrated on the first few hundred items? Okati et al. show that confidence ignores whether the expert is actually better on an item, so it spends budget where escalation does not help. A prefix-calibrated cutoff also has no way to respond when the mix of hard items changes partway through the stream. In random order this alternative may come close. The burst and late-surge orders are designed to show whether it does not, and we will report the random-order result even if the gap there is negligible.
Why not just train Differentiable Triage offline and freeze it? This is the real threat. If the deployment stream looks like the historical data, a frozen threshold is near-optimal and our online machinery adds nothing. Our claim is limited to shifted or bursty arrival and to settings where expert loss must be learned from deferred items only. The comparison is run under both stationary and shifted orders so that the scope of any advantage is explicit. If frozen offline triage matches us under shift too, the proposal's central claim fails.
Why not just pool all expert reviewers into one flat tier? Pooling removes the interaction between tiers, and it may well be as good when tiers differ only in capacity. Guo et al. (Embodied LLM Agents Learn to Cooperate in Organized Teams) found a 9.76% task-completion-time gain from designated leadership in GPT-3.5 teams. But that was in household simulation and measured coordination speed, not accuracy, so it is not evidence for our setting. The merged-tier ablation decides the question directly. We expect the hierarchy to help only when senior tiers are measurably more accurate on a narrow slice of items, and we will report the accuracy gap between tiers so readers can see when that condition holds.
Proposed Experiments
The first benchmark is the Twitter Hatespeech data used by Okati et al.: 24,783 tweets, each labeled by multiple annotators. Tier 2 is a single randomly drawn annotator, tier 3 is the majority of the remaining annotators, and the ground truth is the full majority.
The second is Galaxy Zoo, with 10,000 images classified by volunteers. Tiers are formed analogously from small versus large volunteer subsets.
The third is PeerRead (A Dataset of Peer Reviews (PeerRead)). It includes rejected as well as accepted submissions, which makes it a closer stand-in for desk triage than the accepted-only FIRSTPASS (FIRSTPASS). Tier 1 is a classifier on paper text. Tier 2 is one reviewer's recommendation. Tier 3 is the mean of all reviews. The ground truth is the final decision. As anchors for tier 1, PeerRead reports 65.3% acceptance accuracy on ICLR and 75.7% on cs.CL with a 22-feature model. FIRSTPASS reports 80.5% accuracy for a fine-tuned Qwen2.5-7B on its STANDARD-versus-EXTENDED task. We use FIRSTPASS only as a secondary stream for that task, and acknowledge its survival bias.
Metrics are final system error, budget utilization per tier, regret against the hindsight oracle as a function of stream length, and the bias of upper-tier loss estimates. Each order type is run over 20 random seeds.
Three checks decide the outcome. First, under random order, regret should shrink as the stream lengthens. Second, under burst and late-surge orders, the pacing term should cut error relative to the frozen offline baseline. Third, removing exploration should produce a measurable upward bias in tier-3 loss estimates that degrades escalation decisions. If the second check fails, the online formulation is unnecessary in practice.
As a secondary condition, we replace tier 2 with an LLM given the PeerRead review criteria in three prompt formats. This tests whether the pacing rule absorbs prompt-induced instability or amplifies it.
References
- Differentiable Learning Under Triage
- Optimal Streaming Algorithms for Submodular Maximization with Cardinality Constraints
- Prof-K: Probabilistic One-Pass Filtering for Efficient Top-k Selection
- Beyond 1/2-Approximation for Submodular Maximization on Massive Data Streams
- Learning and modeling big data
- Policy-as-Prompt: Rethinking Content Moderation in the Age of Large Language Models
- Embodied LLM Agents Learn to Cooperate in Organized Teams
- A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes