Refereed Pseudo-Labels for Per-Scene Diffusion Adaptation

Recent 3D editing methods remove an object from a captured scene and fill the hole convincingly from every viewpoint. They do this by fine-tuning a pretrained image generator on the scene itself, at test time. A diffusion model is a generator that learns to turn random noise into images by denoising it step by step. These pipelines generate their own training targets: inpainted views, warped views and pseudo-labels. They then fine-tune on those targets without anyone checking them. This is the equivalent of an author refereeing their own paper. When the generator hallucinates a wrong structure, fine-tuning locks that structure in across all views. Two failures are especially common: the removed object reappears faintly, or a background edge bends where the warping went wrong.

We propose to borrow the structure of a journal editorial board and apply it to this self-training loop. Every generated training target is a "submission." Several independent, specialized referees check it: a geometric check, a photometric check and a semantic check. None of the referees is the model being adapted. An editor then makes a non-compensatory decision to accept, revise or reject. Non-compensatory means a single failed check cannot be outweighed by excellence on the others. A rejected submission is regenerated with a new random seed and conditioned on neighbouring regions that were already accepted. It gets a bounded number of rounds before its region is handed to the 3D representation's own smoothing and regularization. The claim is concrete: on 360-degree object removal, gating test-time adaptation targets this way reduces geometric artifacts and reappearing objects more than the scalar losses and scalar confidence filters in current use. It does so at a cost comparable to the hour-per-scene these pipelines already spend.

Where this sits

The adaptation side supplies the method and the failure. IMFine removes Gaussians from a 2D Gaussian Splatting scene. It then predicts and smooths depth for the hole, inpaints one reference view, warps it to the other views, and adapts a diffusion refiner at test time on augmented scene images. The pipeline is open-loop. An error in the depth estimate flows straight into the warped targets, and nothing checks the refiner's outputs. Our method reuses IMFine's pipeline wholesale and inserts review between generation and adaptation.

SPIn-NeRF keeps its 2D inpainter (LaMa) frozen. It handles disagreement between views by training the radiance field with a perceptual loss, which is a soft arbiter that averages rather than decides. LoRA3D comes closest to the review idea. It down-weights geometrically inconsistent pseudo-labels using a closed-form robust-estimator rule before fine-tuning low-rank adapters on a geometric foundation model. That filter has one criterion (reprojection consistency), it is compensatory, and it has no revise step. Our proposal differs in all three respects. SAP3D protects its adapted model with a retrieval-based prior-preservation loss. It motivates our concern that the adapted model drifts toward whatever it trains on.

The governance side supplies the design principles, not the method. Journal front matter such as IEEE Transactions on Information Theory describes the familiar tiering: an editor-in-chief sets policy, associate editors route work, and referees evaluate in their own specialty. Principal-agent analysis such as Dong explains why delegating evaluation to the party being evaluated invites moral hazard. The multi-agent LLM literature gives two cautions. Liu's incident-response study found that decentralized teams of heterogeneous specialists deadlock: 12/20 successes against 14/20 for centralized teams with a single captain. This argues for a single editor with a fixed decision rule rather than negotiation among referees. Guo et al. and Mindstorms show that hierarchical roles help language-model teams, and Mindstorms applies them to 3D generation. Both work by prompting black-box models, and Guo et al. document the communication overhead. We therefore deliberately implement the board as cheap numerical checks, not LLM agents.

Core Claim

In per-scene diffusion adaptation, the quality of what the model trains on matters more than how it is trained. Confirmation bias in self-training comes from targets that pass one criterion while failing another. A scalar score lets strength on one criterion hide failure on another. A veto rule with a revise loop does not. The new ingredients are threefold: referees that are independent of the adapted model, non-compensatory acceptance, and targeted regeneration of rejected regions.

Research Program

The pipeline is IMFine with a review stage. Each generated target is a candidate inpainted region in one view, and three referees score it.

The geometry referee compares the target's depth with the rendered depth of the post-removal Gaussian scene. The target's depth comes from a monocular depth estimator distinct from the one IMFine uses upstream. Where the filled background is actually observed in other views, the referee also checks reprojection agreement.

The photometric referee measures two things. The first is seam consistency with the unmasked pixels of the same view. The second is agreement of colour statistics with already-accepted targets that cover overlapping 3D regions.

The semantic referee uses frozen image features to measure two things. One is how well the filled region matches its surrounding context. The other is similarity to crops of the removed object, which catches the object reappearing.

Each referee needs a threshold, and we calibrate it per scene without ground truth. Borrowing the idea that referees need known standards, we cut synthetic holes in observed background regions of the scene, run the same inpainting path, and record each referee's score against the true error. Each referee's threshold is set at a fixed false-accept rate on these known holes. The editor then applies one rule. A target is accepted only if every referee passes, rejected if any referee fails by a large margin, and otherwise sent for revision. Revision means resampling with a new seed, with neighbouring accepted targets used as conditioning, for at most $K$ rounds (we will try $K \in \{1,2,3\}$).

Tiering keeps the cost down. The cheapest check, seam consistency, runs first as a desk-reject screen, so the expensive referees only see survivors. Only accepted targets enter test-time adaptation. Regions that remain rejected after $K$ rounds are supervised only by the 3D representation's regularizers.

The analysis side asks two questions. First, which referee's vetoes carry the gains? Second, does the calibrated false-accept rate on synthetic holes predict the error that reaches the final render? If it does, the calibration procedure is useful on its own.

Baselines

The naive reference point is IMFine as published: every generated target is used, with no review. On the IMFine benchmark it reports a PSNR of 19.67, against 18.75 for SPIn-NeRF (IMFine). We rerun it with its released settings and time budget, so all comparisons use our own reproduced numbers alongside the reported ones. SPIn-NeRF serves as the frozen-inpainter reference, representing soft arbitration through a perceptual loss.

The strongest competitor is a compensatory filter. It combines the same three referee scores into one weighted confidence, and it also uses a LoRA3D-style robust re-weighting of each target's loss by that confidence (LoRA3D). It sees exactly the information our editor sees. We give it the same compute, and we tune its weights with the same synthetic-hole calibration data, using a hyperparameter search budget equal to the one used for our thresholds. Beating it is the result that matters.

Two ablations isolate the key ingredients. The veto-without-revision ablation rejects failing targets but never regenerates them; this isolates the effect of the revise loop. The revise-with-scalar-score ablation regenerates targets that score low on the weighted sum; this isolates the effect of non-compensatory gating. We count as a win a consistent improvement over the tuned compensatory filter on LPIPS and on the reappearance and artifact measures below, across the majority of scenes, while PSNR does not fall.

Why not just…?

Why not just fold the three checks into the training loss as extra weighted terms? This is the most serious alternative, and the compensatory baseline tests it directly. Our argument is that a weighted sum lets a sharp, photometrically seamless reappearance of the removed object score well overall. The semantic check alone does not punish it enough to outweigh the other two. We will stratify scenes by failure type. If tuned weights match the veto rule even on the scenes where reappearance and bent edges occur, the contribution shrinks to "multi-criteria filtering helps," and we will report exactly that.

Why not just draw N samples per view and keep the best under one score? Best-of-N is cheap and strong, and the revise-with-scalar-score ablation matches it at equal sample counts. The difference we expect is in conditioning. Revision draws on already-accepted neighbours, so it samples near a consistent solution rather than independently. If independent resampling at the same number of diffusion calls matches our method, the conditioning buys nothing.

Why not just use a vision-language model as critic, in the style of Mindstorms? Mindstorms runs its critiques as prompted dialogue among black-box models (Mindstorms). Guo et al. report that elaborate deliberation raises communication cost (Guo et al.), and Palla et al. show that decisions from language-model policies shift with prompt formatting (Policy-as-Prompt). For thousands of targets per scene, we need a critic that is deterministic and calibrated. We will still run a small side comparison on a few scenes, substituting a VLM for the semantic referee and measuring agreement with our reappearance detector and the added time.

Proposed Experiments

We will use the 20-scene IMFine benchmark, measuring PSNR, LPIPS and FID. The file gives no LPIPS or FID values, so for those metrics there is no reported result on Lacuna to compare against, and we will measure all methods ourselves.

We will also use the SPIn-NeRF object-removal dataset, whose captures include the true background and so allow grounded error measurement. Its reported number is 91.66% segmentation IoU. Lacuna has no reported IMFine PSNR on this dataset.

We add two targeted metrics. The first is a reappearance rate: the fraction of rendered test views in which the removed object is detected in the filled region. The second is depth error in the hole, measured against a dense reconstruction of the object-free capture on SPIn-NeRF. We also report wall-clock time on the same GPU, since IMFine is roughly one hour per scene on a V100. The review stage should add no more than about a third to that.

Three checks will decide whether the idea holds. The first is that our method beats the tuned compensatory filter on reappearance rate and hole depth error without losing PSNR. The second is that removing the revise loop recovers part of the gain and removing the veto recovers most of it; this tells us which ingredient matters. The third is that, across scenes, the false-accept rate measured on synthetic holes predicts the final render error on real holes, which would validate per-scene calibration.

A negative result on the first check against the compensatory filter would be informative. It would show that the useful part of the editorial idea is independent checking, not how the verdicts are combined.