How Good Can a Tree-Crown Detector Get? Measuring the Human Ceiling in Extreme-Environment Benchmarks

In places like tropical rainforests and debris-covered Himalayan glaciers, there is no practical way to walk out and measure the "true" answer. The labels used to train and score machine learning models are drawn by experts looking at drone or satellite images. When those experts would disagree with one another, a model's score against any single expert's labels mixes two things: the model's own errors, and the gap between that expert and the others. Nobody has measured that gap for these benchmarks. This proposal would measure it.

The plan has three parts. First, have several experts independently re-label the same parts of two extreme-environment benchmarks. Second, compute how much they agree after correcting for agreement that happens by chance. Third, score current models on that same human scale. The central claim is that some headline results may already sit near the level of human agreement. The two results in question are the 48.60 crown-detection score on SelvaBox and the 35.94% IoU on debris-covered ice, where IoU (intersection over union) is the fraction of overlap between predicted and labeled areas. If they sit near human agreement, progress on these benchmarks is being measured with a ruler that has run out. The proposal also separates two kinds of expert disagreement: "is there a crown here at all?" versus "where exactly does its edge run?" It then asks whether model uncertainty can see the first kind.

Core Claim

SelvaBox is a set of centimeter-resolution drone mosaics of tropical forest. Five biologists spent over 1,200 hours drawing 83,137 crown boxes on it. The authors score detectors with RF1₇₅, a per-image F1 that counts a match only when a predicted box and a labeled box overlap by at least 75% IoU. They write that "perfect scores are unlikely due to inherent ambiguity in the imagery," but they never quantify that ambiguity. Boundary Aware U-Net for Glacier Segmentation treats a single map of debris-covered ice from ICIMOD (the International Centre for Integrated Mountain Development) as infallible, even though debris on ice looks spectrally identical to the rock moraines around it.

Our claim is that both benchmarks need a measured human-agreement ceiling. For dense crown detection, that ceiling has to be chance-corrected. Agreement statistics such as Cohen's kappa subtract the agreement two raters would reach by luck. In a closed canopy packed with similarly sized crowns, a box placed at random will often overlap a real crown well, so raw inter-annotator F1 overstates how much the experts really share.

What is new is not the idea of measuring agreement. That was done for skin-lesion masks in Handling Inter-Annotator Agreement for Automated Skin Lesion Segmentation, which found a median kappa of about 0.72 and called the result an "accuracy ceiling." What is new is three things:

  1. a chance-corrected agreement statistic for object detection in dense scenes, where the usual pixel kappa does not apply;
  2. putting models and humans on the same agreement scale for benchmarks where no physical ground truth can ever be collected;
  3. testing whether the "silent" uncertainty failure reported in medical imaging carries over to Earth observation.

Research Program

Collecting the repeated labels. We would draw a stratified sample of SelvaBox tiles across its Brazil, Ecuador and Panama sites and across canopy density classes. Five independent experts would each label every sampled tile without seeing the existing annotations. One of them would re-label a subset weeks later, which gives a within-rater baseline. The original effort works out to roughly 70 boxes per hour. At that rate, about 5,000 crowns labeled five times costs on the order of 350 annotator-hours, roughly a quarter of the original budget.

We would repeat this on the glacier side. Four or five glaciologists would independently re-trace debris-covered and clean ice on a sample of the held-out grid cells of the Hindu Kush Himalaya Landsat 7 dataset, the test split used in the glacier paper.

A chance-corrected detection statistic. For two annotators A and B and an overlap threshold $t$, let $P_o(t)$ be the F1 between their box sets at threshold $t$. Let $P_e(t)$ be the expected F1 when B's boxes are randomly moved within the tile's canopy area. The relocation keeps B's box count, box sizes and non-overlap structure, so that $P_e$ captures what dense packing alone produces. The detection agreement is then

$$\kappa_{\text{det}}(t) = \frac{P_o(t) - P_e(t)}{1 - P_e(t)},$$

which has the same form as Cohen's kappa. It equals 1 for identical labels and 0 for agreement no better than shuffled boxes. With five raters we pool pairwise terms in the style of Krippendorff's alpha, which handles incomplete overlap between raters. For glacier masks, the standard pixel-level Cohen's kappa and Krippendorff's alpha apply directly.

Crucially, we compute the same statistics for each model against each human. A model then counts as "at ceiling" when its agreement with the experts falls inside the spread of expert-to-expert agreement.

Splitting disagreement into presence and boundary. We cluster all annotators' boxes into candidate crowns, then record two quantities for each crown:

  • presence disagreement: how many of the five experts marked it at all;
  • boundary disagreement: among those who marked it, how much their boxes differ, measured by the spread of IoU between them.

This follows the distinction made in Beyond Boundary Noise. On lung CT, that paper found that entropy from Monte Carlo dropout or ensembles detected radiologist presence disagreement no better than chance, with an AUROC of 0.48. A small probe trained on the network's frozen internal features did much better, reaching an AUROC of 0.74. Their result motivates our test but does not settle it. Ambiguity in forests comes from crowns physically interleaving, not from faint nodules, and it could behave differently.

The glacier analysis follows the same pattern. Presence disagreement means debris ice versus bedrock; boundary disagreement means where the debris margin ends.

Baselines

The do-nothing baseline: current single-annotator scores. This is the reference every claim is compared against. On SelvaBox, the DINO detector with a Swin-L backbone reaches RF1₇₅ = 48.60 on the internal test set and 41.91 zero-shot on BCI50ha, where Detectree2 reaches 34.97 (SelvaBox). On glaciers, the SLBA loss reaches 35.94% IoU on debris-covered ice and 68.17% on clean ice, against 29.05% for standard Dice loss (Boundary Aware U-Net).

We would retrain or reuse these exact models. Every model is scored on the same re-annotated tiles, against each expert separately and against the original labels, with no extra tuning. The finding counts as meaningful only if the human-human interval differs reliably from the model-human interval in either direction, judged by bootstrap over tiles.

The strongest existing ambiguity detector: the post-hoc probe on frozen features. This is the Ambiguity Head from Beyond Boundary Noise, a small network with attention pooling that reads the frozen internal features of the segmentation model. We port it to the frozen features of the DINO detector and of the glacier U-Net. It is trained only on presence-disagreement labels from training sites and evaluated on held-out sites. It competes against ensemble entropy and detector confidence score, using the same detector checkpoints and the same tuning budget for each.

No reported result was found on Lacuna for presence-ambiguity detection on any Earth-observation dataset. The only reference numbers are 0.74 and 0.67 AUROC on lung CT, which set the order of magnitude to expect. We count it as beating entropy if it gains at least 0.1 AUROC on held-out sites.

Ablation 1: drop the chance correction. Here we report raw inter-annotator F1 alone. This isolates whether the correction changes any conclusion.

Ablation 2: geometric smoothing before measuring agreement. We apply morphological opening and closing and convex hulls to the glacier masks before measuring agreement, the operations tested on skin lesions in Handling Inter-Annotator Agreement. This separates disagreement caused by hand jitter while tracing from real disagreement about what is there.

Why not just…?

Why not just report raw pairwise RF1 between annotators and skip chance correction? This is the most serious threat, because raw human-human RF1₇₅ is already a usable ceiling. We expect the two measures to diverge mainly at loose overlap thresholds and in the densest canopy, where shuffled boxes match often. At the strict 75% threshold, chance matches may be rare, and correction may barely move the numbers. If $P_e$ turns out negligible across thresholds, we will say so and recommend the simpler raw measure. Both are computed from the same data, so the experiment settles this directly.

Why not just lower the overlap threshold, say to RF1₅₀, since 75% may be too strict for crowns? Choosing a threshold blindly replaces one arbitrary choice with another. We sweep the threshold from 0.3 to 0.9 and plot human-human and model-human agreement curves together. The right threshold is where the experts themselves stop agreeing, and that is an empirical output of this study rather than a guess.

Why not just use ensemble uncertainty to find the ambiguous crowns and skip the expensive re-annotation? On lung CT, entropy scored 0.48 AUROC for presence ambiguity (Beyond Boundary Noise). However, that is a different modality, and forests may differ. Our multi-annotator labels are exactly what is needed to test this. If entropy works well on canopy, that would be a useful and cheaper result, and we would report it.

Why not just have a senior expert adjudicate disputed crowns? Adjudication yields one cleaner label but still hides how much the experts disagreed, and that disagreement is what bounds achievable accuracy. Beyond Black & White found that collapsing votes into a single label lost useful information on ambiguous items, though only in text-tagging tasks. We keep the adjudicated label as one more reference to score models against.

Proposed Experiments and Decisive Checks

The first decisive check is the ceiling comparison. Where does DINO's chance-corrected agreement with each expert fall relative to the expert-to-expert distribution, on the internal sites and on held-out sites? The same comparison applies to SLBA's debris IoU against the glaciologists' pairwise IoU. If the models already fall within the human spread, then the gap between 48.60 and 41.91 should be partly re-read as differences in labeling convention between sites.

The second check is whether model rankings change. We re-rank DINO, Faster R-CNN and Detectree2 by human-normalized agreement and test whether the order shifts in dense tiles.

The third check is the presence-ambiguity test: AUROC and Spearman correlation between each uncertainty signal and the number of experts who marked each crown, under a leave-one-site-out evaluation.

The fourth check is whether presence and boundary disagreement can be told apart at all on glaciers. Geometric smoothing should raise boundary agreement but leave presence disagreement unchanged, which is the pattern seen on skin lesions. If smoothing removes most of the disagreement, the glacier ceiling is a tracing-jitter problem rather than a problem of perceiving what is there, and the proposal's emphasis would shift accordingly.