Gates That Know When Experts Would Disagree

Flow and mass cytometry measure dozens of proteins on millions of single cells. Labs still identify cell populations by "gating": an expert draws polygons on a sequence of two-marker scatter plots, for example keeping live cells, then CD3-positive cells, then splitting those into CD4 and CD8. Two experts given the same file often draw different gates and report different cell counts (UNITO; Bayesian Trees). Automated gating tools now match expert consensus closely on aggregate numbers. They still treat that consensus as the single truth. They give one answer per gate and no signal about which gates a human should check.

We propose to keep the gate itself as the explanation and change how it is placed. Each gate stays a 2D region on a named marker plot, so a clinician can read it and edit it as usual. A high-capacity model decides where to draw that region, using the whole sample as context. The model is trained on each expert's gates separately rather than on their consensus, so for every gate in every sample it predicts how far experts would plausibly diverge. Gates with low predicted divergence are drawn automatically. Gates with high predicted divergence go to a human, and a human edit to one gate updates every gate below it in the hierarchy. The test is not agreement with consensus. It is whether the automated counts fall within the range that real experts produce, and how many human edits it takes to get there.

Core Claim

The accuracy-interpretability trade-off in automated gating comes from mixing up two things: what the user inspects, and what information the model uses. In our design the inspected object is fixed by construction. It is a 2D gate in the lab's own hierarchy, and it can be corrected the way Concept Bottleneck Models allow concepts to be corrected. Only where the gate is placed depends on a learned summary of all markers. If the claim holds, three things follow:

  1. Using full-marker context to place gates should improve per-sample accuracy without making the gates any less readable.
  2. Disagreement-aware training should identify the gates where real experts diverge better than generic model uncertainty does.
  3. Deferral combined with edit propagation should reach expert-level agreement with fewer human edits than correcting a strong automated gater after the fact.

Relation to Prior Work

The closest work is UNITO (Automated cytometric gating with human-level performance using bivariate segmentation). It turns each gate's parent population into a 101×101 density image and segments it with a U-Net, applying gates in hierarchy order. Its masks are already 2D gates. Our proposal therefore does not claim "gates as concepts" as new.

We differ from UNITO in three ways:

  • Context for placement. UNITO places a gate using only the two plotted markers. We allow the placement to depend on the joint distribution of all markers, while the gate shape stays 2D.
  • Training target. UNITO trains toward a single consensus. We train toward individual experts and output a predicted spread.
  • Human interaction. UNITO has no deferral rule and no propagation of human edits through the hierarchy.

Our method draws on two other lines of work. From Concept Bottleneck Models we take the idea that corrections to intermediate concepts should change downstream outputs. From DeCoDe we take concept-level routing between machine-only and human decisions. DeCoDe routes using a separate learned selector and was tested only on image benchmarks. We route using predicted inter-expert spread on cytometry gates, which is a different signal.

Two papers constrain the design:

  • Concept Bottleneck Models show that side channels bypassing the bottleneck undermine interpretability. This is why our marker context may move a gate but never changes its 2D shape or the cell's membership rule within a sample.
  • Bayesian Trees adapt gate thresholds per patient using a shared template (its random-effects extension). It is the strongest interpretable competitor, but its cuts are axis-aligned and its inference is slow.

Several methods are not interpretable at the gate level and appear only as flat-label comparisons:

Research Program

The gate model. Take a gate $j$ with parent gate $p(j)$ and plotted marker pair $(a_j, b_j)$. For sample $s$, let $u = (x_{a_j}, x_{b_j})$ be a cell's coordinates on that plot. Let $z_s$ be a set-encoder embedding of all parent-population cells over all markers. The gate head outputs

$$q_{j,k}(u \mid z_s) = \Pr[\text{expert } k \text{ includes the cell in gate } j \mid \text{cell is in } p(j)],$$

with one output per training expert $k$ and a shared trunk across experts. Because $z_s$ is fixed within a sample, each expert-specific gate is a region in the 2D plane that can be drawn and edited. The predicted spread for gate $j$ is the across-expert variance of the predicted population fraction,

$$\sigma^2_{j,s} = \mathrm{Var}_k\Big[\tfrac{1}{|P_s|}\textstyle\sum_{i \in P_s} q_{j,k}(u_i \mid z_s)\Big],$$

where $P_s$ is the set of parent-population cells in sample $s$. A gate is deferred when $\sigma_{j,s}$ exceeds a tolerance set to match the observed inter-expert spread on validation data.

Edit propagation. When a human replaces gate $j$ with their own polygon, the cells in each child gate change. The model recomputes $z_s$ for those children and redraws them. This is the cytometry version of a test-time concept intervention.

Evaluation harness. We will build a small, shared evaluation harness, in the spirit of the uniform fit/predict interface in UCEE and the decoupled components in DeepDIVA. It exists to keep the comparison fair, not as a contribution in its own right. Every method:

  • receives the same transformed events, with an arcsinh cofactor fixed per dataset, because gating results are sensitive to the choice of transform (flowVS);
  • receives the same gating hierarchy and the same expert labels;
  • must return per-gate regions, an optional spread estimate, and a response to an edit.

Methods without gate structure report only flat per-population metrics.

Metrics. The evaluation replaces single scalars with four quantities:

  • Within-envelope rate. The fraction of (sample, gate) pairs whose predicted population frequency lies inside the range of the held-out experts' frequencies. This is needed because a cross-subject correlation of 0.98 can coexist with large per-sample errors.
  • Rare-population F1. F1 on populations below 1% of parent events.
  • Deferral AUROC. How well the deferral signal ranks the gates where held-out experts actually disagree beyond tolerance.
  • Edits-to-agreement curve. Per-sample frequency error as a function of the number of gates a simulated expert has edited.

Baselines

Do-nothing reference. A single expert's gates are scored against the other experts. This is the human noise floor. Any automated system whose within-envelope rate matches it has reached "human-level" in the sense that matters here. A second naive reference applies one expert's template gates unchanged to every sample, which measures how much per-sample adaptation is needed at all.

UNITO, the strongest existing method. It is the closest structure to ours, and it reports an average correlation with a four-immunologist consensus of 0.98 on mass cytometry and 0.97 on flow cytometry. Under synthetic Gamma noise, its F1 stays near 0.9 (UNITO). Fair-run conditions:

  • We run it with its 101×101 density maps and convex-hull post-processing.
  • It is trained on consensus labels on the same training samples, under the same hyperparameter-trial budget and GPU hours as our model.
  • Its deferral signal is the mean per-pixel entropy of its mask.
  • Its intervention is a human replacing its mask for the edited gate before downstream gates are rerun.

Beating it means matching its correlation within noise while achieving a higher within-envelope rate, higher deferral AUROC, and a lower area under the edits-to-agreement curve.

Bayesian Trees with random effects, the strongest interpretable competitor. It reports 96.9% accuracy on the AML CyTOF benchmark and 92.3% on BMMC (Bayesian Trees). We run it on the same hierarchy with its informative prior table. Its posterior spread over cut positions serves as its deferral signal.

Flat classifiers. DeepCyTOF reports mean F-measure ≥ 0.98 across 208 CyTOF samples. On the multi-center batch dataset it reports 0.985 with calibration and 0.925 without (DeepCyTOF). optimalFlow reports mean F-measure 0.9539 on Euroflow (optimalFlow). These are compared only on flat per-population F-measure. The aim is to show that keeping gate structure costs little or nothing in accuracy.

Ablations. Each isolates one ingredient:

  • Placement context. The context $z_s$ is restricted to the parent's 2D density, which is UNITO-like input inside our architecture. This tests whether full-marker context helps.
  • Disagreement training. Expert-specific heads are replaced by a five-member ensemble trained on consensus, with spread taken from ensemble variance. This tests whether training on individual experts matters.
  • Propagation. Propagation is switched off, so an edited gate's children stay as originally predicted.

No reported result was found on Lacuna for within-envelope rate, deferral AUROC, or edits-to-agreement for any cytometry method. The baselines' numbers on these metrics will therefore be produced inside the harness.

Why not just…?

Why not just use UNITO and let the expert fix whichever masks look wrong? This is the most serious threat. UNITO's gates are already editable 2D regions. If an expert scanning all gates is as fast as our deferral, our contribution reduces to the evaluation protocol. The UNITO baseline above runs exactly this workflow. If its edits-to-agreement curve matches ours when experts are guided by its own mask entropy, we will report that and drop the method claim.

Why not just a concept bottleneck whose concepts are marker positivity (CD3+, CD4−)? In cytometry such a concept is a threshold on a single input channel, so the concept layer would be nearly the input itself. All of the subjectivity lies in where the threshold sits for a particular sample. That placement is what our gate heads learn. It is also what the axis-aligned per-patient cuts in Bayesian Trees approximate, and it is included as a baseline.

Why not just use model uncertainty, or a consensus-trained ensemble, to decide what to defer? This could work. Uncertainty about the consensus gate and expert disagreement are related, but they are not the same. A crisp density valley can still be gated differently by experts who follow different lab conventions, and a consensus model would be confident there. The ensemble ablation tests this directly. If ensemble variance matches the deferral AUROC of expert-specific heads, the simpler ensemble wins and we will say so.

Why not just add more markers to the gate input and let the network gate in high dimensions? That gives up the drawable, editable 2D region that clinicians verify. It would also reopen the leakage problem described in Concept Bottleneck Models. Our design confines extra markers to placing the gate. The context ablation measures how much accuracy is lost by imposing that restriction.

Proposed Experiments

Data. The key requirement is individual expert gates on shared files. The UNITO cohorts were gated by at least four immunologists: the COVID-19 vaccine CyTOF cohort with 40 subjects, the acute COVID-19 cohort with 23, and the young-versus-old flow cohort with 97. We will seek the per-expert gates for these cohorts. If only the consensus is available, we will have four experts independently gate a subset of samples under our own protocol.

Further datasets cover flat comparison and batch shift:

  • the AML and BMMC CyTOF benchmarks;
  • Euroflow, which spans four centers;
  • FlowCAP-I;
  • the DeepCyTOF multi-center batch set of 16 samples from one subject across instruments, used to train on one instrument and test on the others.

Protocol. We use leave-one-expert-out evaluation. For each fold, the model trains on three experts' gates. The held-out expert, together with the observed spread among all four, defines the target range. In the edits-to-agreement simulation, the held-out expert acts as the oracle: when a gate is deferred, it is replaced with that expert's polygon.

Decisive checks. There are four, each tied to part of the core claim:

  1. Accuracy. Our within-envelope rate should exceed UNITO's, and no flat metric should be worse than UNITO's reported correlation band.
  2. Context. The full-context model should beat the 2D-only ablation on rare-population F1 and on the cross-instrument split. If it does not, the extra marker context is unnecessary.
  3. Deferral. Expert-specific heads should beat both the consensus ensemble and UNITO mask entropy on deferral AUROC.
  4. Human effort. With propagation, the edits-to-agreement curve should reach the human noise floor using fewer edited gates than both the no-propagation ablation and the UNITO edit workflow.

Each check can fail independently. Each failure would narrow the claim to the part that survives.