Amodal segmentation means predicting the full outline of an object, including the parts hidden behind other objects. Real annotations for hidden parts are scarce and subjective, so the field increasingly trains on rendered scenes, where the full outline comes for free. A second line of work fills large missing regions step by step instead of all at once. These progressive refiners grow a mask outward from confident evidence, and they avoid the blurry, incoherent guesses that single-pass models make deep inside a void. Combining the two looks natural: train a progressive refiner on synthetic data and apply it to real images. It has a hidden flaw. A refiner only ever sees its own intermediate states, and on synthetic data those states are clean. On real images the first guess fails in real-specific ways, such as truncated edges on glass, masks bleeding into reflections, or missing thin parts. The refiner then treats those failures as trusted anchors and builds on them.
We propose to fix this by training the refiner on corrupted intermediate masks whose errors are made to look statistically like the first-pass model's errors on unlabeled real images. The corruption is learned adversarially, borrowing the logic of point-dropout masking from LiDAR domain adaptation. In two sentences: the sim-to-real gap for a progressive mask refiner lives mostly in the distribution of its intermediate mask errors, not in raw appearance. If you teach the refiner to correct realistic real-domain mistakes, which you can synthesize without real labels, it corrects instead of compounding them.
Core Claim
The closest existing system, EUREKA from Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer, trains an encoder-only vision transformer on rendered retail scenes. It predicts amodal masks in one pass, with attention restricted to a union of the visible and predicted regions, and reaches 37.10% amodal AP on the real Real-IRAIS test set, against 14–19% for earlier methods. Its authors report failures under severe occlusion and on translucent or reflective packaging. The progressive-refinement literature explains why single passes struggle with large voids. Image Inpainting for Irregular Holes Using Partial Convolutions shrinks the invalid region layer by layer. Onion-Peel Networks for Deep Video Completion fills only the outer ring of a hole at each stage. Recurrent Feature Reasoning for Image Inpainting does the same inside feature space. The same literature also warns that iterative loops lock in early errors.
Our claim has two parts. First, a lightweight refiner can raise zero-shot real-domain amodal AP over a strong single-pass model. The refiner operates on masks, geometry, and pooled features, grows the amodal mask outward from a soft confidence map, and is trained on adversarially realistic corrupted masks. Second, the benefit over simple hand-crafted mask noise should concentrate where real errors are content-dependent: glass, reflective plastic, and heavy occlusion. On opaque, moderately occluded objects we expect hand-crafted noise to be nearly as good. That stratified prediction is what makes the claim falsifiable. It is also what separates the learned corruption from the cheap alternative.
Research Program
The system has three parts. The first is a first-pass amodal predictor $f$, either EUREKA or a reproduction, frozen after synthetic training, which maps an image to an initial amodal mask $m_0$ per instance.
The second is a progressive refiner $R$. It runs $K$ steps (we expect three to five). At each step it carries a soft per-pixel confidence map, a continuous version of the validity mask in partial convolutions, and updates only a band of pixels adjacent to the currently confident region. It uses mask-adaptive gated convolutions of the kind introduced by Efficient Indoor Depth Completion Network Using Mask-adaptive Gated Convolution. That work showed gates conditioned on local hole shape beat uniform partial-convolution handling and still run at 24.4 FPS. We reuse the operator, not the depth task. Keeping $R$ convolutional and mask-centric is deliberate. Its inputs are the visible mask, the current amodal estimate, the confidence map, and low-resolution backbone features. Masks are far less domain-specific than raw RGB, so the appearance gap reaches $R$ mainly through the errors in its input masks.
The third part is a learned corruptor $g$, and it is the new ingredient. Adversarially Masking Synthetic to Mimic Real (ASM) makes clean synthetic LiDAR resemble real LiDAR by learning which points to drop. A Gumbel-Softmax mask, meaning a differentiable way to sample keep-or-drop decisions, is trained against a domain discriminator. This lifts SynLiDAR→SemKITTI from 15.2% to 22.8% mIoU. ASM corrupts inputs at a single shallow layer. We instead corrupt the intermediate state of the refinement loop. On synthetic data, $g$ takes the ground-truth amodal mask $y$ together with renderer metadata and outputs a corrupted mask $\tilde{y}$. The metadata includes per-object material class, depth, and occlusion order, which pipelines such as MaviGen and SynTable already export. A discriminator $D$ tries to tell $(\tilde{y}, v)$ on synthetic images from $(m_0, v)$ on unlabeled real images, where $v$ is the visible mask. The objective is
$$\min_{g}\max_{D}\; \mathbb{E}_{\text{real}}[\log D(m_0, v)] + \mathbb{E}_{\text{syn}}[\log(1 - D(g(y, a), v))] + \lambda\, \mathcal{L}_{\text{budget}}(g(y,a), y),$$
where $a$ is renderer metadata, and $\mathcal{L}_{\text{budget}}$ penalizes corruption larger than the first-pass model's measured error rate on synthetic validation data, so $g$ cannot win by destroying masks. $D$ sees only mask geometry and never pixels. This is ASM's lesson applied here: a discriminator given appearance would separate the two domains on rendering artifacts, not on error structure.
$R$ is trained on synthetic data to map both $\tilde{y}$ and its own rolled-out intermediate states back to $y$, with a per-step loss. The error distribution the refiner learns to fix therefore comes from the real domain, while supervision stays synthetic. Real images are used only unlabeled, through $D$.
Two analyses accompany the build. The first measures error compounding directly. For each real test instance we track IoU with ground truth after every refinement step, and ask whether training on corrupted masks turns a declining curve into a rising one. The second tests whether $g$ has learned something real. We compare the spatial statistics of $g$'s corruptions with the true errors of $m_0$ on a small labeled real subset: area error, boundary distance, and concentration on glass or reflective material.
Baselines
The naive reference is EUREKA trained on 3D-IRAIS and applied zero-shot. Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer reports 37.10% amodal AP on Real-IRAIS for it. This is also the strongest existing method on the question, since it is the only reported sim-to-real amodal result in the file with a real test set of this kind. We will reproduce it with the authors' protocol and report both our reproduction and the published 37.10%. All gains are measured against our reproduction on identical data. Beating it means a gain over the reproduced EUREKA larger than seed-to-seed variation across three training seeds. We target at least two AP points overall, with the larger gain expected on the material-hard subset.
The second baseline is the same refiner $R$ trained with teacher forcing on clean synthetic intermediate masks and no corruption. It isolates whether progressive refinement alone helps or, as we predict, partly hurts on real data through compounding.
The third is $R$ trained on hand-crafted corruptions: random dilation and erosion, boundary jitter, and random cuts along the occlusion boundary. The noise budget is tuned on synthetic validation data with the same number of tuning trials as $g$. This is the ablation that isolates the proposal's key ingredient, and the one most likely to threaten it.
The fourth is $R$ trained with $g$ but without renderer metadata. It tests whether content-conditioned placement of errors matters.
The fifth is a strong single-pass model with extra capacity: EUREKA with added transformer layers matching $R$'s parameter count and compute. It checks that gains do not come from size.
For the tabletop setting, the reference is UOAIS-Net trained on SynTable-Sim. SynTable reports ACC_oo of 82.9 for it on OSD-Amodal, and gives no numeric Overlap or Boundary F-measure. We will therefore rerun it to obtain those numbers. All methods share the backbone, synthetic training data, unlabeled real pool, and GPU-hour budget.
Why not just…?
Why not just perturb intermediate masks with hand-crafted noise? This is the most serious alternative, and it may well capture most of the gain on opaque rigid packaging. The file gives no direct evidence against it. Our argument is that real errors are content-dependent. EUREKA's reported failures cluster on glass and reflective surfaces and under heavy occlusion, and noise spread uniformly over masks cannot put its errors there. The third and fourth baselines settle the question on a stratified test. If hand-crafted noise matches $g$ on the material-hard subset too, we will report that the gain comes from exposure to noise during refinement training, and drop the adversarial component from the claim.
Why not just self-train on pseudo-labels from real images? Self-training feeds $m_0$ back as a target. On exactly the instances where $m_0$ is systematically wrong, such as truncated glass bottles, it reinforces the error. This is the compounding problem moved from inference into training. Our method uses real images only to learn what the errors look like, and keeps the correction targets synthetic and exact. We will run a standard confidence-thresholded self-training baseline on the same unlabeled pool and compare it within each stratum. The two methods are also complementary, and we will test them stacked.
Why not just run a diffusion prior or multi-sample voting at test time? MaskComp shows that voting over masks from several diffusion samples removes individual hallucinations: FID falls from 30.8 for Stable Diffusion 2.1 to 16.9 on AHP. AISDiff extracts cross-attention shape priors, meaning the diffusion model's internal maps of where each prompt word lands, over ten timesteps, and reaches 36.9 amodal AP on KINS. pix2gestalt reaches 82.87 zero-shot mIoU on COCO-A. None of these was evaluated under a sim-to-real split. All three need multi-pass diffusion inference, which their own papers flag as too slow for real-time use. pix2gestalt also hallucinates physically wrong poses. We will run AISDiff-style priors on Real-IRAIS as a quality-versus-latency reference point, not as a like-for-like competitor, and report frames per second next to AP.
Why not just add more domain randomization to the renderer? The IRAIS authors already use photorealistic procedural scans, and earlier models trained on them stayed at 14–19% AP. ASM shows that clean synthetic sensors leave a structural shortcut that texture and lighting randomization do not remove. Randomization changes appearance. It does not tell the refiner what its own real-domain mistakes look like.
Proposed Experiments
The primary experiment trains on 3D-IRAIS, uses unlabeled real retail images for $D$, and evaluates on Real-IRAIS by amodal AP. For the unlabeled pool we will use D2SA training images with their labels discarded. D2SA and Real-IRAIS are distinct real retail datasets, so there is no transductive leakage. Results are stratified by occlusion rate and by a material tag (glass or reflective versus opaque) that we annotate on Real-IRAIS instances. The secondary experiment trains on SynTable-Sim, uses unlabeled real tabletop images, and evaluates on OSD-Amodal with Overlap F-measure, Boundary F-measure, [email protected], and ACC_oo. This tests whether the effect holds outside retail.
Three results would decide the proposal. First, the per-step IoU curve on real data must rise for the corruption-trained refiner and must not rise for the teacher-forced refiner. If both rise, the compounding story is wrong. Second, on the glass and reflective stratum, $g$ with metadata must beat hand-crafted noise and $g$ without metadata by a margin larger than seed variance. If it does not, the adversarial component is unnecessary. Third, corruptions produced by $g$ must match real first-pass error statistics better than hand-crafted noise does, measured on the small labeled real subset. Latency is reported throughout. The refiner should add under a third to single-pass inference time, a target set by MagaConv's real-time operator budget, and an order of magnitude less than diffusion-based alternatives.
References
- Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer
- Image Inpainting for Irregular Holes Using Partial Convolutions
- Onion-Peel Networks for Deep Video Completion
- Recurrent Feature Reasoning for Image Inpainting
- Efficient Indoor Depth Completion Network Using Mask-adaptive Gated Convolution
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- SynTable: A Synthetic Data Generation Pipeline for Unseen Object Amodal Instance Segmentation of Cluttered Tabletop Scenes
- Completing Visual Objects via Bridging Generation and Segmentation
- Amodal Instance Segmentation with Diffusion Shape Prior Estimation
- pix2gestalt: Amodal Segmentation by Synthesizing Wholes