Is the Elevation Doing the Work? Leakage-Controlled Ablation of Tree Crown Segmentation

Drone-based tree crown segmentation means drawing an outline around every individual tree in an aerial image. It feeds biomass estimates, mortality maps, and reforestation audits. The field's recent progress comes largely from adding elevation data, a digital surface model (DSM) giving the height of the canopy top at each pixel, to ordinary color imagery. The reasoning is physical: two touching crowns of the same species look like one continuous patch of green, but a height dip between them should reveal the boundary. The reported gains are real but modest, a few points of mean average precision. They are measured on image tiles cut from the same few orthomosaics, where neighboring tiles share trees, lighting, and plantation grid layout.

This note proposes a test of whether those gains come from the model reading 3D crown geometry, or from memorizing site-specific layout that leaks between spatially adjacent training and test tiles. The method pairs two tools borrowed from other fields. The first is spatially blocked evaluation splits, which remove leakage between neighbors. The second is counterfactual edits to the elevation input, which tell us what the network actually uses. The output is a corrected estimate of how much elevation contributes. It also yields a reusable stress test built from real forest data, which does not become stale when a benchmark's tiles are trained on.

Core Claim

We claim that part of the headline accuracy, and possibly part of the DSM gain, in current crown segmentation benchmarks is an artifact of spatial autocorrelation. Evaluation tiles are near-copies of training tiles, so a model can succeed by recalling a site's spacing, row orientation, and height statistics rather than by separating crowns from height troughs. The honest measure of elevation's contribution is not "RGB+DSM minus RGB" on a random tile split. It is that difference measured on a spatially blocked split, and then checked against elevation edits whose effect on a structure-reading model is predictable in advance.

The closest prior work, Bringing SAM to new heights and its companion Assessing SAM for Tree Crown Instance Segmentation from Drone Imagery, runs additive modality ablations across boreal, temperate, and tropical sites. The comparisons there are SAM variants, a DSM-encoder model called BalSAM/DSMPrompter, and Mask R-CNN with and without DSM. That work motivates this proposal: it establishes that DSM helps. It does not erase or corrupt the DSM, and it does not control for leakage between adjacent tiles. We do both, on the same models and data.

Research Program

The work has three parts. Each borrows a technique from outside forestry.

Part 1: leakage-controlled splits. This transfers the lesson of Blindfold Baselines for Embodied QA, which found that most test questions also appeared in training and that a question-only bag-of-words model reached 50.34% accuracy. It also transfers the artifact-stripping logic of Back to Square One.

For each orthomosaic in the Quebec Plantations (26,428 tiles), Quebec Temperate, and Barro Colorado Island (BCI) sets, we rebuild the evaluation split in three increasingly strict ways. The first is the published split. The second is a spatially blocked split: square blocks with a buffer at least one maximum crown diameter wide, so no crown straddles train and test. The third is a leave-one-orthomosaic-out split. Any crown cut by a tile boundary is assigned to exactly one side.

A two-point loss when moving from the random to the blocked split on a model reporting a 4.5-point DSM gain changes the story completely. That is why this part comes first.

Part 2: counterfactual elevation edits. This follows the feature-erasure logic surveyed in Analysis Methods in Neural Language Processing: if removing an input leaves the output unchanged, the model does not depend on it. Crown geometry lets us go further than plain erasure, because the right behavior of a structure-reading model can be predicted for each edit. We apply five edits at test time only, holding the RGB image fixed:

  • Blindfold. DSM set to a constant. This is the analogue of the question-only baseline.
  • Swap. DSM taken from a different tile of the same site. This keeps the site's height statistics but removes geometry specific to the tile.
  • Trough-filling. Height dips along ground-truth boundaries between adjacent crowns are raised to the lower crown's rim. A model reading geometry should merge those crowns more often.
  • Trough-deepening. Those dips are lowered. The same model should split crowns more cleanly, or at least not get worse.
  • Per-crown vertical offset. Each crown's height is shifted by an independent constant. Boundaries are preserved but absolute heights are destroyed. A model using relative structure should be roughly invariant; a model keying on site-typical absolute heights should break.

Pairing trough-filling with trough-deepening is the key ingredient. Ordinary corruption confounds "uses the input" with "is disturbed by noise," whereas these two edits push predictions in opposite directions only if the network reads boundaries.

Part 3: a cheap factorial. This addresses the combinatorial ablation cost noted in the component-analysis literature. We do not run all $2^k$ combinations. We cross three factors: split strictness, training modality (RGB, DSM, RGB+DSM), and test-time edit. All other components stay at published settings.

We summarize with one quantity, the structure credit:

$$C = \big[s(\text{RGB+DSM}) - s(\text{RGB})\big]_{\text{blocked}} - \big[s(\text{RGB+DSM}) - s(\text{RGB+DSM, swap})\big]^{-}_{\text{blocked}}$$

Here $s$ is the segmentation score, and the bracket with superscript minus is the shortfall from the swap edit. That shortfall is the portion of the gain that still survives when the model is given a plausible but wrong elevation map, and so could not be coming from geometry specific to the tile. Specifically, $[\cdot]^{-}$ is the part of the fused model's score retained under swap beyond what the RGB-only model achieves, floored at zero.

Baselines

The naive reference point is RGB-only Mask R-CNN trained and evaluated on the same splits. On Quebec Plantations, Bringing SAM to new heights reports 60.32 weighted mAP (wmAP) for Mask R-CNN. On BCI the same paper reports 5.52 multi-class mAP without DSM and 8.30 with it. We retrain these from the released configurations under each split, so every model sees identical tiles, epochs, and augmentation. We use the same hyperparameter search budget per split, tuned on a validation block that is itself spatially separated from test blocks.

The strongest existing method is BalSAM/DSMPrompter, which pairs a frozen SAM image encoder with a trainable DSM encoder. It reports 64.84 wmAP (±1.49) on Quebec Plantations Bringing SAM to new heights; the companion paper Assessing SAM for Tree Crown Instance Segmentation from Drone Imagery reports the same figure for DSMPrompter. It is the strongest competitor because it is the model whose advantage is attributed explicitly to elevation. "Beating" it here does not mean a higher score. It means showing that its advantage over RGB-only either survives our blocked split with positive structure credit $C$, or shrinks by more than its reported ±1.49 spread. The latter would be evidence that the published margin is partly leakage.

Two structure-only references bound what elevation alone can do. The first is classical canopy-height-model peak-finding with watershed growth, which uses no learning and cannot memorize a site. The second is SAM prompted with DSM local maxima, reported to raise mIoU from about 35% to over 50% Bringing SAM to new heights. If a learned fused model loses more than these references when moving to blocked splits, its extra accuracy came from memorization, not geometry.

For RGB-only transfer we include DeepForest, a RetinaNet pre-trained on 30 million LiDAR-derived crowns, which reports 72% recall and 64% precision on the NEON benchmark. We also include the DINO/Swin-L detector from SelvaBox, which reports raster-level F1 of 48.60 at 75% IoU internally and 41.91 zero-shot on BCI50ha. For spatially blocked versions of any of these benchmarks, no reported result was found on Lacuna, so those numbers are the new measurements.

The key ablation is the trough-fill/trough-deepen pair run on the fused models. It isolates whether boundary height drops, the proposed mechanism behind elevation's value, are what the network uses.

Why not just…?

Why not just use spatial block cross-validation, which remote sensing already knows about? Blocked splits are necessary, and they are Part 1. They are not sufficient. A blocked split can show that absolute scores drop but cannot say why the DSM helps in what remains. A fused model could still exploit per-site height statistics that transfer across blocks within one orthomosaic. The swap and per-crown-offset edits detect exactly that. If blocked splits alone collapse the DSM gain to zero, the edits become moot, and we will report that plainly. It would be the simplest and most useful outcome.

Why not just measure cross-site zero-shot transfer, as SelvaBox does with BCI50ha? Cross-site drops mix leakage with genuine domain shift: different sensors, resolutions, species, and annotators. Bringing SAM to new heights itself notes that the 1 m BCI elevation blurs small crowns, so a transfer loss cannot be attributed to memorization. Within-site blocked splits hold sensor and species fixed, isolating adjacency. We report cross-site transfer as a complementary axis, not a substitute.

Why not just inspect saliency maps on the DSM channel? Attribution maps show where gradients are large, not whether the output depends causally on boundary geometry. Heatmaps of this kind are prone to the "speculation disguised as explanation" criticized in Troubling Trends in Machine Learning Scholarship. The opposite-direction trough edits are an intervention with a predicted sign, which saliency cannot provide.

Why not just evaluate on synthetic forests with perfect ground truth, such as QC-SF? QC-SF shows a large synthetic-to-real gap: 91.66% mIoU on synthetic test data against 46.39–52.12% on real Montmorency plots. Synthetic tests would therefore measure fidelity to the simulator, which is the procedural-quirk risk noted for dynamic benchmarks. Our edits keep real imagery and real crowns and alter only one controlled property. That is how we carry the idea of generated, uncontaminated test items from DyVal into sensory data.

Proposed Experiments and Decisive Checks

We retrain RGB Mask R-CNN, RGB+DSM Mask R-CNN, and BalSAM under the three split regimes on Quebec Plantations, Quebec Temperate, and BCI, with three seeds each. We apply the five test-time edits to every fused model and run the peak-finding and maxima-prompted SAM references on identical test blocks. On NEON we additionally check whether DeepForest's LiDAR pre-training plots overlap evaluation rasters spatially, and rerun evaluation excluding overlapping areas. Metrics are wmAP and mAP as in the SAM papers, raster-level F1 at 75% IoU for SelvaBox comparability, and merge/split counts per crown pair for the trough edits.

Four outcomes are decisive:

  1. Leakage. If the RGB-only Mask R-CNN score drops by more than the seed spread when moving from the published to the blocked split, adjacency leakage inflates the benchmark.
  2. Genuine elevation value. If BalSAM's margin over RGB-only on blocked splits stays at or above roughly 4.5 wmAP and survives the swap edit, the published elevation claim holds and we have strengthened it.
  3. Mechanism. If trough-filling raises merges while trough-deepening lowers them, the network reads boundary geometry. If both leave merges unchanged while the swap edit also barely hurts, the elevation gain is a site-statistics shortcut.
  4. Robustness check. Per-crown offsets separate relative-structure models from absolute-height ones; this is the check most likely to expose why models degrade on coarse or differently calibrated DSMs.