Measuring Hidden Detail: Occluded-Contour Metrics for Amodal Segmentation

Amodal segmentation asks a model to outline the whole of an object, including the parts hidden behind something else. The field has recently accepted that ordinary overlap scores are misleading here, because visible pixels dominate them. Benchmarks now score the hidden region separately. What they still measure is how much area a prediction covers inside that region, and not whether its shape is right. A model can fill the hidden part of a bicycle wheel with a smooth rounded blob and still score reasonably on occluded-pixel overlap, even though it never recovered a single spoke. There is also a second failure that current metrics cannot see. A model may draw a confident edge inside the hidden region simply because the object in front has an edge there. In that case the hidden object's outline is "detailed," but the detail belongs to the occluder.

This note proposes an evaluation protocol that scores the contour of the hidden portion of an object. The protocol has three parts: precision and recall of the hidden outline, a separate score for edges copied from the occluder, and averaging across how much of each object is visible. We validate it on ground truth whose hidden shapes are known exactly. We then use it to answer a mechanism question that the architecture literature has left open: do shortcut connections that pass high-resolution features around a network's bottleneck help or hurt fine detail behind occluders?

Core Claim

Visibility-aware metrics and detail-preservation metrics have been built separately, and each is blind where the other looks. On the visibility side, Real-VAS introduced IoU on occluded pixels only (mIoUocc) and IoU on fully occluded frames (mIoUffo). Peek-a-Boo introduced Average Precision Hidden, which scores plane masks behind furniture. WALDO introduced Unbiased Average Recall, which averages recall uniformly over visibility deciles. All three are area-overlap or recall scores. None of them asks whether a boundary is in the right place.

On the detail side, gen2seg shows that keeping a generative model's full decoder recovers thin structures: 51.4 mIoU on iShape against 16.8 for the Segment Anything Model. Its evaluation, however, covers visible instances only, and the authors report failures under egocentric occlusion.

Our claim is that a metric restricted to the hidden portion of the ground-truth outline, together with an explicit occluder-leakage term, will rank existing amodal models differently from mIoUocc. It will also reveal a specific failure mode, copying edges from the occluder, that area metrics reward rather than penalize. The protocol is the main contribution. A controlled architecture study is the first scientific use of it.

The Metric

Four regions define the metric. The visible mask of an object is $V$, its amodal ground-truth mask is $A$, and the hidden region is $H = A \setminus V$. The occluder's visible edge pixels are $E_{occ}$.

Hidden contour recall $R_H$ is the fraction of ground-truth contour pixels of $A$ lying in $H$ that have a predicted contour pixel within a tolerance of $d$ pixels. Hidden contour precision $P_H$ is the fraction of predicted contour pixels in $H$ that fall within $d$ of the ground-truth contour. We combine them as $F_H = 2P_HR_H/(P_H+R_H)$. This is the standard boundary F-score, restricted to the hidden band. To limit sensitivity to where the visible part ends and the hidden part begins, we exclude a strip of width $d$ around the visible–hidden junction.

The new term is leakage:

$$ L_H = \frac{|\{p \in \hat{C}_H : \mathrm{dist}(p, E_{occ}) \le d,\ \mathrm{dist}(p, C_H) > d\}|}{|\hat{C}_H|}, $$

where $\hat{C}_H$ is the set of predicted contour pixels inside $H$ and $C_H$ is the set of ground-truth contour pixels inside $H$. In words, $L_H$ is the share of predicted hidden-region edges that sit on the occluder's edges but not on the true hidden outline.

Following WALDO's reasoning, both $F_H$ and $L_H$ are computed within each visibility decile and averaged uniformly across deciles. This keeps lightly occluded instances from dominating the average. We report results at several values of $d$ to show how sensitive the scores are to the tolerance.

The metric is useful only if it is valid. The first experiment therefore checks that $F_H$ separates the true amodal mask from degraded versions that are matched to have nearly equal mIoUocc: Gaussian-blurred versions, convex-hull versions, and "prior-template" versions. It also checks that $L_H$ detects synthetic leakage when occluder edges are injected into a prediction. If $F_H$ cannot separate these cases more sharply than mIoUocc does, the metric does not add enough to justify itself.

Research Program

Evaluation needs hidden outlines that are known exactly. We draw them from two sources.

The first is Real-VAS. It composites real video footage to produce pixel-perfect amodal masks and has an occlusion prevalence of 65.0% and a mean occlusion severity of 60.0%.

The second is a controlled set we build ourselves, iShape-Occ. We composite real occluders onto iShape, a benchmark of wires, branches and other thin structures, following the Real-VAS compositing strategy. Because the hidden part was visible in the original image, its outline is known exactly. The hidden structures are also exactly the kind a blob cannot fake. Compositing leaves seams that a model could exploit, so we blend the paste edges. We also report results with occluders drawn from a separate held-out source.

For multi-layer stacks we also evaluate on Intra-AFruit, ACOM and MUVA. The file reports these only at the 64×64 resolution used by Sequential Amodal Segmentation. We therefore treat them as a coarse-resolution check where boundary tolerances must be scaled down, not as a detail benchmark.

The mechanism study addresses a tension the literature raises but has not tested. SEIS measures how well a network's intermediate features track spatial position. It shows that U-Net shortcut connections restore this position tracking after the bottleneck erases it, but it tests only rotations, translations and scaling, never occlusion. At hidden coordinates, however, the early features describe the occluder. ProtoCar had to mask its loss so that gradients flow only through visible features, which prevented occluder texture from corrupting hidden vehicle geometry. That is indirect evidence that ungated feature routing hurts.

We train one amodal U-Net on iShape-Occ in several variants: full shortcuts, no shortcuts, and shortcuts gated by the predicted visible mask. In the gated variant, shortcut features at hidden coordinates are zeroed, and the decoder must draw detail there from upsampled bottleneck features alone. Our prediction is as follows. Full shortcuts will show high $L_H$. Removing shortcuts will lower $L_H$ but collapse $F_H$ toward blob-level values. Gating will sit between the two. The unknown worth measuring is whether any variant recovers hidden thin structure much better than a blob does, or whether hidden detail is essentially absent from current architectures.

Baselines

The naive reference points involve no learned amodal completion. The first predicts the visible mask alone. The second predicts the visible mask plus a convex-hull or template completion. These define the floor of $F_H$ and show how much mIoUocc a model earns without recovering shape. Any learned model that fails to beat the template floor on $F_H$, while beating it on mIoUocc, is evidence for the core claim rather than against it.

The strongest existing amodal method on real video in the file is Diffusion VAS. Real-VAS reports 0.604 overall mIoU for it and 0.501 for TABE. TCOW is also included because Real-VAS reports it has an advantage on containment mIoUocc and mIoUffo. No per-model mIoUocc figure is given in the file, so we recompute mIoUocc, mIoUffo, $F_H$ and $L_H$ for all three from their released outputs or checkpoints on identical test splits. For the multi-layer data, we include the cumulative-mask diffusion model from Sequential Amodal Segmentation, which reports 92.6 AP on Intra-AFruit against 78.9 for PLIn.

We also adapt gen2seg to amodal prediction, since it is the strongest visible-boundary model. On visible thin structures its reported iShape mIoU is 51.4. Evaluating its hidden contours tests whether visible detail skill carries over behind occluders.

The ablation that isolates the key ingredient is the shortcut study above, plus one further variant: full shortcuts with the visible mask supplied as an extra input channel. That variant gives an ordinary network the information it would need to learn gating on its own. All variants share the same backbone, input resolution, data, number of training steps and hyperparameter search budget. Differences are reported over three seeds.

"Beating" the metric baselines means something specific here. The metric must reorder model rankings relative to mIoUocc in a way that a human judgment study of hidden contours confirms. For the architecture variants, a real improvement means a gain in $F_H$ larger than seed variance, with no increase in $L_H$.

Why not just…?

Why not just report mIoUocc at stricter thresholds? A stricter overlap threshold still scores area. A blob that covers the hidden region of a branch can clear a high IoU threshold when the branch is thick, and it can fail for reasons unrelated to shape when the branch is thin. InScope shows the same symptom in 3D tracking: accuracy collapses at IoU 0.7 without revealing why. The matched-mIoUocc validation experiment answers this question directly. If thresholded mIoUocc separates blurred ground truth from true ground truth as well as $F_H$ does, we will adopt the simpler metric.

Why not just apply the standard boundary F-score to the whole amodal mask? The visible outline usually makes up most of the contour, so a whole-mask score inherits exactly the visible-region dominance that Real-VAS documented for mIoU. We report the whole-mask boundary F-score alongside $F_H$. The proposal holds only if the two diverge on real models. If they do not, the hidden-band restriction is unnecessary.

Why not just give the network the visible mask as an input and let it learn to ignore occluder features? This is the real threat to the gating hypothesis. It is included as an ablation for that reason. If the mask-channel variant matches gating on both $F_H$ and $L_H$, we will report that explicit gating is unnecessary. The protocol's value does not depend on gating winning.

Why not just super-resolve a coarse amodal mask? The authors of Sequential Amodal Segmentation suggest super-resolution as future work for their 64×64 outputs. The gap between gen2seg and the Segment Anything Model on iShape suggests that thin structures lost at low resolution are hard to restore afterward. That evidence comes from visible segmentation only, however. We test the idea directly by upsampling the low-resolution model's masks with a learned upsampler and scoring $F_H$.

Decisive Checks

The proposal rests on three results. First, $F_H$ must separate true hidden outlines from area-matched degraded versions better than mIoUocc does, on both iShape-Occ and Real-VAS. Second, rankings of Diffusion VAS, TABE, TCOW and adapted gen2seg under $F_H$ must differ from their rankings under mIoUocc. Third, $L_H$ must be measurably nonzero for full-shortcut models and must respond to the gating and mask-channel ablations. If the first check fails, the metric should be abandoned. If the first holds but the second fails, the metric is correct but redundant in practice. The third check determines whether the architectural question has a real answer.