Amodal segmentation asks a vision model to outline the whole of an object, including the parts hidden behind other objects. Examples are the full silhouette of a car behind a truck, or a person behind a fence. Benchmarks score these predictions the way they score ordinary segmentation: by pixel overlap (intersection-over-union, IoU) with one human-drawn outline. Half of that outline is a guess, because the annotator could not see behind the truck either. As a result, a single number mixes two very different errors. One is getting a visible edge wrong, which the image can verify. The other is drawing a hidden edge differently from one annotator, which the image cannot verify and which may not be an error at all. There is direct evidence that this matters. In a blind study, humans preferred a model's completed shapes over the human ground truth itself, 46.68% to 39.50% (Variational Amodal Object Completion). Reference-based image metrics have also been shown to reward incomplete amodal completions (Reasoning-Driven Amodal Completion).
We propose to evaluate amodal masks by splitting each object's contour at the occluder. Visible contour segments are scored strictly against the image-grounded reference. Hidden contour segments are scored against two things: hard physical constraints that need no reference, and a small set of independent human completions rather than one. We will first measure, on a new multi-annotator subset of KINS and COCOA, how much human annotators actually disagree on hidden versus visible contours. That measurement decides whether the idea is worth anything. We then test whether the split metric orders existing methods closer to human pairwise preferences than Mask IoU, Boundary IoU, or a multimodal-LLM judge does.
Core Claim
Amodal benchmark scores are confounded in a specific, measurable way. A large share of the error that Mask IoU and Boundary IoU assign along hidden contours sits within the spread of human annotators, so it is not a model failure. Meanwhile two kinds of violation that humans find obviously wrong are only weakly penalized: a "completion" that extends into visible background, where the object would have been seen if it were there, and a prediction that fails to cover its own visible pixels. A metric that separates verifiable from unverifiable contour, and replaces the single hidden-contour reference with a containment check plus a multi-hypothesis comparison, should track human judgment better. It should also stay deterministic, cheap, and computed directly on masks.
The closest existing pieces each cover part of this. Boundary IoU shows that interior pixels hide contour errors and restricts IoU to a band around the contour. It was built for visible-only segmentation and treats every contour pixel identically, so applied to amodal masks it penalizes hidden-contour deviations more sharply, not less. MAC-Score from Reasoning-Driven Amodal Completion uses a vision-language model to judge completeness and consistency of RGB completions. It reaches a Spearman correlation of 0.516 with humans, against 0.297 for LPIPS. It is non-deterministic, costly per image, and judges synthesized appearance rather than masks. Amodal Ground Truth and Completion in the Wild removes annotator guesswork by projecting 3D meshes into MP3D-Amodal. That only works where scans exist, and on KINS and COCOA the paper reverts to single-reference mIoU. Our proposal differs from all three because it operates on masks, uses the scene's own occluders as a reference-free constraint, and treats hidden-contour ground truth as a distribution estimated from several annotators.
The Metric
For one object, let $V$ be its visible mask, $O$ the union of its occluders, $A$ the predicted amodal mask, and $H = A \setminus V$ the predicted hidden part. The occluders are the visible masks of instances in front of it, from KINS occlusion ordering or COCOA pair labels, plus the image border to handle truncation. The reference-free term penalizes hidden area placed outside any occluder and visible pixels left uncovered:
$$C(A) = 1 - \frac{|H \setminus O| + |V \setminus A|}{|A|}.$$
Here $C(A)$ is 1 when every hidden pixel lies behind something and every visible pixel is covered. The visible-contour term is Boundary IoU computed only over reference contour segments that do not border $O$. We keep the 2% image-diagonal band recommended by Boundary IoU. The hidden-contour term compares the prediction with each of $K$ independent human completions $G_1,\dots,G_K$ inside the occluder region:
$$S_{\text{hid}}(A) = \max_k \, \text{BIoU}_{O}(A, G_k),$$
where $\text{BIoU}_O$ is Boundary IoU restricted to contour pixels lying inside $O$. Taking the max means a prediction matching any plausible human hypothesis is not penalized. As a control we also report the mean over $k$, which rewards a model for hitting the annotators' average shape. Following the Boundary AP convention of taking a minimum, we aggregate the three terms with a min to get one number. We also always report the three terms separately, because the profile is the main diagnostic.
Research Program
The work has three stages, and each one can stop the project.
The first stage is data. We will select roughly 1,500 occluded instances, split across KINS (driving, rigid cars plus deformable pedestrians), COCOA (everyday scenes), and D2SA (rigid grocery items). Each instance is stratified by occlusion rate, with the bucket above 80% oversampled because prior work reports performance collapsing there (Amodal Instance Segmentation With KINS Dataset; Amodal Panoptic Segmentation). Five annotators will independently draw the amodal mask for each instance. From these we compute inter-annotator Boundary IoU separately on visible and hidden contour segments. If hidden-contour agreement turns out to be nearly as high as visible-contour agreement, the multi-hypothesis term adds nothing and we drop it.
The second stage is human preference collection. For the same instances we collect masks from AISFormer (AISFormer, transformer queries for occluder, visible, and invisible masks), AISDiff (Amodal Instance Segmentation with Diffusion Shape Prior Estimation, diffusion cross-attention as a shape prior), Amodal-VAE (Variational Amodal Object Completion, a frozen shape-prior decoder), pix2gestalt (pix2gestalt, diffusion synthesis thresholded into masks), and a held-out human annotation. Raters see two masks overlaid on the image and choose which outline is the more believable whole object. This is the same protocol style as the Amodal-VAE study, scaled up and spread across datasets.
The third stage is analysis. We compute agreement between each metric and human preference, both per pair and per method ranking, and check whether the method ordering changes under the new metric. The pix2gestalt paper reports physical failures such as reversed vehicle orientation; the containment term catches only the subset of these that spill outside occluders. We will report honestly that orientation errors fully inside the occluder are not caught.
A secondary test checks whether the metric is useful beyond evaluation. The containment term is differentiable if computed on soft masks, so we will fine-tune AISFormer with an added containment penalty and with down-weighted cross-entropy inside occluder regions. We then check whether human raters prefer its outputs at equal or modestly lower AP.
Baselines
The naive reference point is standard Mask IoU and AP against the single dataset annotation, the protocol behind every number on these benchmarks. For calibration, AISDiff reports 36.9 AP on KINS (ResNet-101), 35.93 AP on COCOA-cls, and 71.01 AP on D2SA (AISDiff). AISFormer reports 33.8 AP on KINS (ResNet-50) (AISFormer). For the question this proposal asks, which is how well Mask IoU agrees with human preference on amodal masks, no reported result was found on Lacuna. Our study produces that number. The new metric beats this baseline if its pairwise agreement with human preference is higher, with a bootstrap confidence interval over instances that excludes zero difference.
The second baseline is Boundary IoU and Boundary AP applied unchanged to the full amodal mask (Boundary IoU). It is the strongest geometric competitor because it already fixes the interior-pixel problem. It uses the same 2% band, so the comparison isolates the split and the hypothesis set. The only published evidence of its sensitivity is in the visible-only setting: PointRend improves Mask IoU by 8% over Mask R-CNN on large objects but Boundary IoU by 22%. Beating it means higher human agreement, concentrated on the hidden-contour errors where we predict Boundary IoU is most wrong.
The third baseline is an MLLM judge. We will run the MAC-Score protocol of Reasoning-Driven Amodal Completion with the same Qwen3-VL model, adapted by rendering each mask as an overlay or as the masked object crop. Its reported 0.516 Spearman correlation with humans was measured on RGB completions in HiFi-Amodal, not on masks, so we will re-measure it on our preference set. It is the strongest human-aligned evaluator in the file. We give it the same images and a fixed prompt, and report variance over three seeds. Matching it while being deterministic and roughly free to compute counts as a practical win; exceeding it counts as a scientific one.
The fourth baseline is a learned mask critic in the style of Human Preference Score. That work fine-tuned CLIP on human choices; we will fine-tune a CLIP image encoder on image–mask overlays, trained on half our preference pairs from two datasets and tested on the third. It gets the same preference data our metric's design was informed by.
Two ablations isolate the key ingredients. The first removes the containment term. The second replaces the max over hypotheses with a single reference, which tests whether multiple annotators matter.
Why not just…?
Why not just apply Boundary IoU to amodal masks, or split it into visible and hidden parts without extra annotators? A visible/hidden split using a single reference is in fact our ablation. If it matches the full metric, the multi-annotator data was unnecessary and we will say so. The prior evidence against a single hidden reference is the human-over-ground-truth preference in Variational Amodal Object Completion. Our stage-one measurement of hidden-contour disagreement tests that directly instead of assuming it.
Why not just use an MLLM judge like MAC-Score? It is the strongest human-aligned evaluator available, at 0.516 versus 0.297 for LPIPS (Reasoning-Driven Amodal Completion). However, it was validated on appearance completions, it varies run to run, and it cannot supply a training signal. If it outperforms our metric on mask preference, the honest conclusion is that our metric is a cheap diagnostic, not a replacement. The containment term would still remain useful as a training penalty, which the MLLM judge cannot be.
Why not just train a preference model on human mask judgments? This is the real threat, since Human Preference Score shows such critics work for generated images. The same paper also documents the problem we expect here: a critic can be optimized against and can drift from physical constraints. Our cross-dataset test, which trains on two datasets and tests on the held-out third, will show whether a learned critic transfers. If it transfers as well as our metric, we will recommend using it alongside our containment term, which is interpretable and cannot be gamed by spilling hidden area into visible background.
Why not just use 3D-projected ground truth such as MP3D-Amodal? Exact masks remove annotator noise, but only for scanned indoor scenes (Amodal Ground Truth and Completion in the Wild). We use MP3D-Amodal as a control rather than a rival. There, the containment term and exact-reference IoU should agree. If they disagree, either our occluder extraction or the containment assumption is wrong.
Decisive Checks
Three results would settle the proposal. The first is the stage-one gap: hidden-contour inter-annotator Boundary IoU should be clearly lower than visible-contour agreement, and the gap should widen with occlusion rate and for pedestrians versus cars. If no gap appears, the multi-hypothesis term is dropped and the contribution reduces to the containment term. The second is human agreement: the split metric should beat Mask IoU and whole-mask Boundary IoU in pairwise agreement with human preference, with the gain coming mainly from instances above 50% occlusion. The third is method ranking: we will check whether the ordering of AISFormer, AISDiff, Amodal-VAE, and pix2gestalt under the new metric differs from their AP ordering, and whether that new ordering matches the human ordering. If rankings are unchanged, the community's existing leaderboards are vindicated, and that is still worth publishing. The fine-tuning experiment is a bonus. It succeeds if human raters prefer the containment-trained AISFormer at a cost of under one AP point on KINS.
References
- Variational Amodal Object Completion
- Reasoning-Driven Amodal Completion: Collaborative Agents and Perceptual Evaluation
- Boundary IoU: Improving Object-Centric Image Segmentation Evaluation
- Amodal Ground Truth and Completion in the Wild
- Amodal Instance Segmentation With KINS Dataset.
- Amodal Panoptic Segmentation
- AISFormer: Amodal Instance Segmentation with Transformer
- Amodal Instance Segmentation with Diffusion Shape Prior Estimation
- pix2gestalt: Amodal Segmentation by Synthesizing Wholes
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference