Statistic-Targeted Delphi Rounds for Annotating Dense Point Patterns

Much of spatial biology and ecology reduces an image to a set of points, such as cell centers or tree crowns, and then asks whether those points cluster, repel each other, or scatter at random. The standard tools answer that question with summary curves. The best known is Besag's $L$-function, which, for each distance $r$, compares how many neighbors a typical point has with what pure randomness would predict. These curves are computed on hand-annotated or machine-detected points. In dense images, trained annotators disagree about where the points are. Five experts labeling cancer cells in the Oriented Cell Dataset reached a Krippendorff's alpha of only 0.760–0.794, and the SelvaBox authors spent over 1,200 hours on tree crowns and still expect imperfect scores "due to inherent ambiguity in the imagery." Nobody has measured how this disagreement propagates into the clustering curves that downstream science actually reports. Nor has anyone built a way to resolve disagreement that concentrates effort where it changes those curves.

We propose to borrow the Delphi method, a structured way of reaching expert consensus through anonymous, repeated rounds with summarized group feedback, and to aim it at the spatial statistic rather than at the annotation. After a first round of independent point annotation, we locate the disputed points: marks that some annotators placed and others did not, or placed far apart. We then rank these disputes by how much each one moves the estimated $L$-function. Only the most influential disputes go back to the panel. Each panelist sees them with the other panelists' anonymized marks and short written rationales, and can revise or justify their own marks. The claim is that this recovers the true clustering structure more accurately per annotator-minute than majority voting, than adding more independent annotators, or than a single adjudicator. It should also avoid the known Delphi failure of converging confidently on a shared wrong answer.

Core Claim

Annotation disagreement in dense point patterns is not uniformly important. A missed isolated cell barely changes second-order statistics. A disputed split or merge of two touching cells changes the count of very short pair distances, and that count is exactly what separates "clustered" from "regular" at small $r$. We therefore expect the useful part of annotation uncertainty to live in a small fraction of disputes.

For a disputed candidate point $c$, we define its influence as the change in the estimated $L$-function when $c$ is included versus excluded from the consensus pattern, integrated over the analysis range: $$ I(c) = \int_0^{r_{\max}} \big(\hat L_{+c}(r) - \hat L_{-c}(r)\big)^2 \, dr . $$ Here $\hat L_{+c}$ and $\hat L_{-c}$ are the edge-corrected estimates with and without $c$, and $r_{\max}$ is the largest interaction distance of interest. The Delphi rounds address disputes in decreasing order of $I(c)$. Rounds stop when the round-to-round change in $\hat L$ falls inside its annotator-bootstrap spread. This replaces the usual "percent agreement" stopping rule with one that is stated in the units of the scientific conclusion.

This is different from the closest existing work in three ways. Metrics Reloaded used a Delphi consortium to decide which validation metric to use, and stopped at pixel, object and segmentation metrics. We use Delphi rounds inside the annotation process, and our target is a point-process statistic. The Human-AI Hybrid Delphi runs consensus on text questionnaire items. Its useful lessons for us are that compact panels of about six senior experts reached thematic saturation, and that preserving conditional rationales beats flattening votes. It never touches spatial data. The Oriented Cell Dataset measures inter-annotator disagreement once and then sets an IoU threshold, with no feedback loop and no point-process analysis of the result.

Research Program

The first component is a disagreement-propagation study, which is worth publishing even if the Delphi part fails. Using existing multi-annotator data and new independent annotations, we match marks across annotators by optimal assignment within a tolerance radius. We call the resulting groups candidate points. We then compute how the spread across annotators in $\hat L(r)$, in the average-nearest-neighbor ratio $R$ (observed mean nearest-neighbor distance divided by its value under randomness), and in persistent-homology Betti curves compares with the effect sizes that papers report. The Betti curves are the topological summaries used by Manrique-Castano et al. to separate glial-scar stages. The motivating question is whether published conclusions of the form "astrocytes cluster at short range" would survive swapping one annotator for another.

The second component is the protocol itself. Round one is independent annotation. Between rounds, software computes candidates, influences and the current consensus pattern. Round two presents the top-influence disputes as image crops, with the panel's anonymized marks overlaid. Each panelist keeps, moves or deletes their own mark and attaches a one-line rationale chosen from or added to a short taxonomy: touching cells, out-of-focus, debris, or occluded crown. Later rounds circulate these rationales, following the controlled-feedback design of classical Delphi. Anonymity is kept throughout because status deference is the pathology Delphi exists to prevent.

The third component is a check for false consensus. Because Delphi rounds pressure outliers, the panel may agree with itself without getting closer to the truth. To detect this, we need ground truth.

Data and Ground Truth

Ground truth for real dense images does not exist, so the core test uses semi-synthetic images. We simulate point patterns with known parameters: homogeneous random patterns, cluster processes, and repulsive Strauss processes. We render them as fluorescence-like nuclei images, matching density, touching-cell frequency and blur to the high-density regime of CellFMCount, whose 3,023 images and 431,321 annotated neural progenitor cells supply realistic appearance statistics. The true $L$-function of each simulated pattern is known exactly.

The real-image tests use the 10 comparison images of the Oriented Cell Dataset, reduced to centroids from its five existing independent annotations. They also use a high-density subset (more than 500 cells) of CellFMCount and dense-canopy tiles from SelvaBox. On real data we cannot measure accuracy. We can measure stability: whether independent Delphi panels on the same image converge to the same $\hat L$.

Panels will be six trained annotators per arm, the size the Hybrid Delphi work found sufficient.

Baselines

The naive reference point is a single annotator's points, with no aggregation. Its error against truth on synthetic images sets the floor. Its spread across annotators on real images shows how much the choice of annotator alone moves the statistic.

The standard practice baseline is independent annotation by the full panel followed by majority merge. A candidate point is kept if more than half the annotators placed it, at the mean of their positions. This is what multi-annotator datasets such as the Oriented Cell Dataset implicitly support, and it costs the same first round as our method. It is the strongest cheap competitor. For reference, the only reported numbers near this question are that dataset's inter-annotator alpha of 0.760 class-aware and 0.794 class-agnostic, and SAM-Counter's count error on CellFMCount of MAE 22.12, with 48.73% of images counted within 5% of truth. We found no reported result on Lacuna for $L$-function or nearest-neighbor-ratio error under annotation disagreement, so the baselines will be measured fresh.

The budget-matched baseline adds independent annotators, instead of Delphi rounds, until total annotator-minutes equal those of our protocol, and then applies the majority merge. The adjudicator baseline has one senior annotator review every disputed candidate, with the same minutes budget. The automated baseline runs a strong detector, SAM-Counter as described in CellFMCount, as an extra independent annotator or as the sole source of points.

Two ablations isolate our key ingredients. Untargeted Delphi re-presents a random sample of disputes of the same size, which tests whether influence ranking matters. Feedback-only Delphi shows the panel median without rationales and without anonymity, which tests whether the Delphi structure matters beyond "look again."

All arms use the same images, the same edge correction and $r_{\max}$ for $\hat L$, the same matching radius, and the same annotator pool randomized across arms. Beating the baselines means lower integrated squared error of $\hat L$ against the known truth on synthetic images at equal annotator-minutes, with bootstrap confidence intervals over images that exclude zero. It also means correct classification of the pattern type (clustered, random or regular at each tested scale) at least as often as the best baseline.

Why not just…?

Why not just take a majority vote over independent annotators? Majority voting treats every dispute alike and cannot fix a systematic shared error. If most annotators merge two touching nuclei, voting preserves the merge. That merge is exactly the error that biases short-range clustering. The Hybrid Delphi results also show that senior minority reasoning is lost under averaging. This is still a real threat when disputes are mostly random noise, and the synthetic experiment settles it directly. We will report the regimes of density and touching frequency where majority merge is already as good as Delphi, and recommend it there.

Why not just hire more annotators with the same budget? Adding independent annotators shrinks random scatter roughly with the square root of their number, because independent errors average out. It does nothing for errors that annotators share. The budget-matched arm tests exactly this. If extra annotators match targeted Delphi at equal minutes, the Delphi rounds are not worth their coordination cost, and we will say so.

Why not just propagate the disagreement as uncertainty, bootstrapping over annotators, and report wider intervals? This is the most serious alternative, because it requires no new annotation. It is honest about scatter, but it assumes the annotators bracket the truth. When they share an error, the interval is confidently wrong. The synthetic experiments will measure how often annotator-bootstrap intervals for $\hat L$ cover the true curve. We will adopt the bootstrap as a reported companion to Delphi rather than a rival wherever its coverage is adequate.

Why not just let a detector like SAM-Counter produce the points? Detectors remove annotator variability but replace it with model bias. CellFMCount reports that fewer than half of SAM-Counter's image counts fall within 5% of truth, and that a staining gap appears across markers. The automated arm measures whether that bias is larger or smaller than human disagreement in $\hat L$ terms.

Proposed Experiments

The decisive experiment is the synthetic comparison: roughly 200 rendered images spanning three process types and three density levels, annotated under every arm with matched budgets. The primary outcome is $L$-function error against truth at equal annotator-minutes. The key check separates convergence from correctness. For each Delphi round we plot the panel's internal spread against its distance from truth. If spread falls while distance does not, the protocol is manufacturing agreement, and we will stop or redesign the feedback.

The second experiment tests whether influence ranking is the active ingredient. It plots error against the fraction of disputes re-presented, for targeted and untargeted ordering. We expect the targeted curve to flatten after a small fraction of disputes, since second-order statistics depend mostly on short-range pairs. If the two curves coincide, the core claim fails.

The third experiment runs two independent Delphi panels per real image on Oriented Cell Dataset, CellFMCount high-density and SelvaBox images, measuring cross-panel agreement of $\hat L$ and of the nearest-neighbor ratio before and after rounds, alongside the existing Krippendorff alpha. A final transfer check reruns the glial-scar-style analysis of Manrique-Castano et al., their Betti-curve stage classifier, on points from each arm. This asks whether annotation choice alone can change a qualitative scientific conclusion.