When human experts draw boxes around tree crowns in a dense tropical canopy, they often disagree. They disagree about where a crown's edge lies, and more fundamentally about whether a tangled mass is one tree or three. Current datasets erase this. The five biologists behind SelvaBox, a drone-imagery dataset of 83,137 tropical crowns from Brazil, Ecuador and Panama, merged their work into single consensus boxes, even though the authors say that finding every tree in such canopy "is subjective." This proposal asks a practical question that anyone building data for underrepresented regions faces. Suppose a fixed number of expert annotation hours can be spent either on getting several independent opinions per image (disagreement) or on single opinions over more images from more sites (diversity). Which choice produces a detector that transfers better to a new country or forest type? And which one tells us more reliably where the detector will fail there?
The core claim is specific. Disagreement collected on a small "anchor" set, together with a record of what kind of disagreement it is (edge extent versus one-tree-or-many), can be used as training-only supervision. We expect it to do two things that extra single-labeled data cannot. It should teach the detector to predict, at unseen sites, where humans would disagree. It should also allow an evaluation that stops penalizing predictions that match a legitimate alternative reading of the canopy. We do not assume the answer on raw transfer accuracy. The budget-matched comparison against diversity is the central experiment, and a loss to diversity on accuracy would be a useful, reportable result.
Why this intersection, and what already exists
Three lines of work set up the question. The first shows that disagreement is information rather than noise. Uma et al. train classifiers on the full vote distribution instead of majority labels and beat training on expert gold labels on CIFAR-10H (66.64% vs 65.22%). Nie et al. show that disagreement distributions reproduce across independent annotator pools. Sharmanska et al. treat disagreement as "privileged information," available only at training time, and use it to soften a classifier's likelihood on hard examples. All three work on categories, not boxes. We reuse the privileged-information framing, and the reproducibility result motivates a test-retest check we build in.
The second line moves uncertainty into spatial outputs, and it constrains our design. He et al. model each box edge as a Gaussian, a bell-shaped distribution with a predicted mean and spread, trained with a KL-divergence loss that down-weights ambiguous edges. On COCO this lifts Faster R-CNN AP from 23.6% to 29.1%. However, the spread is learned against a single consensus box and never sees real human disagreement. Zhou et al. (UAED) do supervise a variance head with real multi-annotator spread for edge detection (ODS 0.829 on BSDS500), and they deliberately up-weight high-uncertainty pixels. That directly contradicts the down-weighting of He et al. and Sharmanska et al. Neither line was tested under geographic shift. Both assume the spread around one answer is unimodal, which the one-tree-or-three case violates. Scarpellini et al. reconcile disagreement among detections of the same scene seen from different viewpoints, not among humans, and do so in simulated indoor scenes.
The third line shows that geographic diversity beats volume. GeoDE reports that adding commissioned multi-region images to ImageNet gains 9% on Dollar Street and 21% on GeoDE. Dollar Street closes a large income gap by fine-tuning on diverse homes. Gustafson et al. trace low-income failures to texture shift, and they note that their factor labels are themselves subjective but aggregate them anyway. These results make diversity the serious competitor. They are all classification, all single-label, and none ask whether a fixed budget is better spent on diversity or on multiplicity. The conceptual critique in The Consensus Trap and the separation of value disagreement from data uncertainty in Separating Value Disagreement from Data Uncertainty motivate our choice to record type of disagreement rather than a single spread number.
Research Program
The first component is data. We re-annotate a stratified anchor subset of SelvaBox tiles covering every orthomosaic and all three countries, with five independent expert annotators per tile working blind to one another. Annotation time is logged per tile, so budgets can be matched in hours rather than in boxes. We would first try to recover the original per-annotator SelvaBox files; if they survive, they supply a free anchor set. A second pool of annotators relabels a fraction of the anchor tiles, so we can measure whether disagreement is reproducible in this domain, as Nie et al. found for language.
The second component converts annotator sets into two kinds of targets. We cluster boxes across annotators per region. Where clusters match one-to-one, the per-edge spread across annotators gives an extent target. Where one annotator's box covers several of another's, the region gets a split/merge label recording the competing groupings.
The third component is the model. We start from the SelvaBox multi-resolution detector. We add a box head that predicts a mean and spread per edge, following He et al., but on anchor tiles the spread is supervised by the empirical human spread, as UAED does for pixels. A separate lightweight head predicts split/merge probability per detection. Off the anchor set, the spread is self-learned as in He et al. At test time no extra annotations are needed.
We then treat the weighting question as an explicit experimental factor rather than resolving it by assumption. We compare three settings: down-weighting high-disagreement boxes in the localization loss, up-weighting them with a progressive schedule in the style of UAED, and leaving them unweighted. We cross this with whether split/merge regions enter the localization loss at all. Our prior is that extent disagreement behaves like UAED's hard boundaries and repays up-weighting. We expect topological split/merge disagreement to behave like the noise Sharmanska et al. guard against, and to be better removed from box regression and routed to its own head. The experiments decide.
Finally, we add a disagreement-aware evaluation. A prediction counts as matched if it matches the grouping chosen by any anchor annotator for that region, scored by the same F1-at-IoU-0.75 matching used by SelvaBox. We report this alongside the standard single-reference score so the two can be compared directly.
Baselines
The naive reference point is the SelvaBox detector trained on consensus boxes only, evaluated with country or site held out. Its reported in-distribution score is RF1₇₅ of 48.60, and its zero-shot score on the Panamanian BCI50ha benchmark is 41.91 versus 34.97 for Detectree2 (SelvaBox). We rerun it under our splits rather than reuse those numbers, because our held-out protocol differs.
The strongest competitor on transfer accuracy is budget-matched diversity. We take the annotation hours spent on the anchor set and instead spend them on single-annotator boxes over additional tiles, drawn preferentially from sites and orthomosaics underrepresented in training. This is the GeoDE lesson applied to detection. It uses the same detector, schedule and hyperparameter search budget, with hours matched from the logs. Beating it means higher held-out-country RF1₇₅, or equal RF1₇₅ together with significantly better disagreement prediction. If it wins on both, our claim fails.
The strongest existing uncertainty method is He et al.'s KL loss with variance voting, trained on consensus boxes with a self-learned spread. It is run with identical backbone and tuning. It isolates whether human spread adds anything over spread the model learns on its own. No reported result was found on Lacuna for KL-loss detectors on tree-crown data; its reported figure is the COCO AP gain from 23.6% to 29.1%.
Two ablations isolate the key ingredient. The first replaces human spread with random box jitter, a synthetic spread matched to the marginal statistics of the human spread, so any gain from real disagreement cannot be explained by generic box-level label smoothing. The second collapses the split/merge label into the extent spread, testing whether recording disagreement type matters.
Why not just…?
Why not just spend the budget on more diverse single-labeled images? This is the real threat, and GeoDE and Dollar Street are strong evidence that diversity drives transfer in classification. We do not argue it away. The budget-matched comparison is the headline experiment, run at several budget levels so the crossover point, if any, is visible. The case for disagreement rests on outputs that diversity cannot produce: a predictor of human disagreement at unseen sites, and an evaluation that separates model error from legitimate ambiguity. If diversity also matches us on the first of these, that tells practitioners in data-poor regions not to pay for multiplicity.
Why not just let the detector learn its own uncertainty, as He et al. do? A self-learned spread measures how hard the model finds a box, not how humans disagree about it. It is also unimodal, so it cannot represent a one-tree-or-three split. The He et al. baseline and the human-versus-self-learned comparison on held-out anchor tiles settle this directly. The test is the correlation between predicted spread and human spread at an unseen country, and the accuracy with which split/merge disagreement is flagged.
Why not just smooth box labels with random jitter? Uma et al. found that real crowd distributions beat uniform smoothing for classification, but that result may not carry over to coordinates. The matched-jitter ablation tests it here. If jitter equals human spread on every metric, the multi-annotator collection is unnecessary.
Why not just use a foundation model zero-shot? The SelvaBox authors report that detectors trained on temperate or generic data do poorly on tropical canopy until trained on tropical data, and Gustafson et al. show a 93% versus 62% zero-shot CLIP gap across income. We include no dedicated foundation-model arm beyond the pretrained backbone, which all methods share.
Proposed Experiments
Transfer is tested in two regimes. The first is leave-one-country-out within SelvaBox: train on two of Brazil, Ecuador and Panama, test on the third, with BCI50ha as an external Panama check. The second is temperate-to-tropical: train on NeonTreeEval and QuebecTrees plus a small anchor set, then test on SelvaBox.
We report four families of metrics. For accuracy, standard RF1₇₅ and RF1 averaged over IoU thresholds 0.50 to 0.95. For ambiguity-aware accuracy, the any-annotator RF1₇₅. For disagreement prediction, the rank correlation between predicted and human per-edge spread, and AUROC for split/merge regions, all at held-out countries. For reproducibility, the test-retest agreement between the two annotator pools on the anchor set.
Three checks are decisive. The first is whether human-supervised spread beats both self-learned spread and matched jitter in predicting human disagreement at an unseen country. If not, the privileged-information transfer claim is false. The second is the sign of the weighting effect for each disagreement type. It resolves the conflict between Zhou et al. and He et al. and Sharmanska et al. in a setting where the stakes are geographic transfer. The third is the budget curve against diversity, which determines whether a lab labeling the next underrepresented forest should buy more opinions or more places. A further check is the gap between standard and any-annotator RF1₇₅. If it is large, some of what the field reports as failure in tropical canopy is disagreement among experts rather than model error.
References
- SelvaBox: A high‑resolution dataset for tropical tree crown detection
- A Case for Soft Loss Functions
- What Can We Learn from Collective Human Opinions on Natural Language Inference Data?
- Ambiguity Helps: Classification with Disagreements in Crowdsourced Annotations
- Bounding Box Regression With Uncertainty for Accurate Object Detection
- The Treasure Beneath Multiple Annotations: An Uncertainty-Aware Edge Detector
- Self-improving object detection via disagreement reconciliation
- GeoDE: a Geographically Diverse Evaluation Dataset for Object Recognition
- The Dollar Street Dataset: Images Representing the Geographic and Socioeconomic Diversity of the World
- Exploring Why Object Recognition Performance Degrades Across Income Levels and Geographies with Factor Annotations
- The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation
- Separating Value Disagreement from Data Uncertainty in Pluralistic Preference Data