Shadow Objects: Per-Scene False-Relation Control for Scene Graph Generation

A scene graph is a compact description of a scene: a list of objects (nodes) and the relationships between them (edges), such as "cup on table" or "chair left of desk." Every scene graph generator in use today decides which relationships to output with a knob chosen once, offline. It keeps the top K scored edges, or it keeps every edge above a threshold tuned on a validation set. Neither choice adapts to the scene in front of it. A cluttered room with 40 objects and a sparse corridor with 6 get the same K. A model moved to a new dataset, or a vision-language model asked about unfamiliar categories, keeps a threshold that was calibrated somewhere else. The result is either missing edges or hallucinated ones: relations that sound plausible but that the sensor data does not support. Downstream reasoners that trust the graph then fail.

We propose to treat "which candidate edges should this scene graph contain" as a variable selection problem, and to borrow a specific tool from the feature selection literature: shadow variables. These are deliberately fake inputs, known to carry no signal, mixed in with the real ones so that the model's scores on them reveal how high noise can score. For scene graphs, the fakes are shadow object pairs. Each one is built from real objects so it looks realistic, but its relation evidence is scrambled, so it cannot contain the relation being scored. How often shadow pairs score above a threshold estimates how many real edges above that threshold are false. Choosing the lowest threshold whose estimated false fraction stays below a user-set target gives each scene its own cutoff, with no labels from the target domain.

Core Claim

Per-scene shadow calibration can hold the realized false discovery rate of emitted relations near a user-chosen target $q$. The false discovery rate (FDR) is the fraction of output edges that are wrong. We expect this to work both in-domain and under distribution shift, and at equal FDR the method should recover more true edges than global thresholds or fixed top-K. Global thresholds miss the target under shift and fixed top-K ignores it entirely.

What would be new is a label-free cutoff that adapts to each scene, applied to the final choice of which edges enter the graph. Prior work prunes candidate pairs before scoring, or re-weights features inside the model. None of it governs the output decision with an explicit error rate.

Where This Sits in Prior Work

The mechanism comes from Direction 1. Probing for sparse and fast variable selection with model-based boosting appends permuted copies of features to the data and stops boosting the first time a copy is selected, which yields a stopping rule from a single run. We reuse that idea but apply it to edges rather than tabular columns, and we count shadow exceedances rather than stopping at the first one. The terminating-random experiments selector fuses many random dummy-variable experiments to hit a target FDR. That supplies our stabilization step: we average over several independent shadow draws per scene. Streaming Feature Selection using IIC motivates why error-rate control matters as the candidate pool grows: fixed criteria either flood with false positives or shut down as dimension grows (BIC admitted nearly 30 false positives per 1,000 features where IIC admitted fewer than 1). The number of candidate edges grows as $N^2$ in the number of objects, so the same pathology is expected when one global threshold is applied to scenes of very different size.

Direction 2 supplies the problem and the constraints. HSGG documents relational hallucination in vision-language models, for example "person in boat" when the person stands beside the boat. It fights hallucination with hand-set geometric pruning plus contrastive decoding against a geometry-only "amateur" model, at about 61 seconds per image. Our method is complementary: it sets how many of the surviving edges to trust, and it adds one extra scoring pass per shadow draw. Edge-Centric Relational Reasoning for 3D Scene Graph Prediction (LEO) adds a learned link predictor that soft-weights candidate edges before reasoning. SceneGraphFusion proposes edges from spatial neighborhoods and re-weights feature channels with attention. Both improve the scores; neither says where to cut them. GroupForward hands a vision-language model a fixed top-K candidate bundle and fails whenever the right object falls outside it. That is exactly the fixed-K brittleness we target.

Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction (OCRL-3DSSG) finds that relation errors track uncertainty in the object classifier. This constrains us in two ways. First, shadow pairs must preserve realistic object-classification uncertainty, or they will be easier to reject than real null pairs and the method will under-report false discoveries. Second, better object encoders are a competing route to fewer false edges, which we must compare against.

Research Program

For each scene with candidate pairs $\mathcal{C}$, we build a shadow set $\mathcal{S}$. A shadow pair keeps the real subject object $i$ and pairs it with an object $j'$ drawn from a different scan. It also replaces the relative-geometry descriptor with one sampled from another candidate pair in the same scene, so the pair's scale and clutter statistics match the host scene. Real features are thus combined with mismatched relational evidence.

Any relation-scoring model scores both sets. This can be a trained 3D predictor such as SceneGraphFusion, or a training-free vision-language scorer in the HSGG style. For each predicate class, the emitted threshold is

$$ \tau_q = \min\Big\{ t : \frac{|\mathcal{C}|}{|\mathcal{S}|}\cdot\frac{1 + \#\{s \in \mathcal{S} : \text{score}(s) \ge t\}}{\max(1,\ \#\{c \in \mathcal{C} : \text{score}(c) \ge t\})} \le q \Big\}, $$

where the first factor rescales shadow counts to the size of the real candidate pool. The whole fraction estimates the share of emitted edges above $t$ that are null. The $+1$ is the standard knockoff-style correction that keeps the estimate conservative when counts are small. We repeat this over $R$ shadow draws and take the median threshold, following the fusion logic of T-Rex.

The analysis rests on one assumption: shadow scores must be distributed like scores of genuinely null real pairs. We test this directly rather than assert it. On labeled 3DSSG we compare the two score distributions per predicate with a two-sample test, stratified by object-classifier entropy (the OCRL-3DSSG concern). When they diverge, we test variants of shadow construction: same-category transplants, geometry-only permutation, and cross-scene transplants. There is a known caveat. 3DSSG labels are incomplete, so some "null" pairs are unannotated true relations. We therefore also audit realized precision by hand on a random sample of emitted edges.

Baselines

The naive reference is the standard protocol: emit a fixed top-K edges per scene (K = 20 and 50, matching the R@K conventions of the 3DSSG literature), or apply one fixed score threshold of 0.5. This is what current systems implicitly do.

The strongest competitor is split conformal calibration: a single threshold computed on held-out labeled source-domain scenes to guarantee target precision. It is principled, cheap, and gives the same guarantee in-domain. It is run on exactly the same scoring model and the same source validation scans that we use for nothing but diagnostics, so it gets strictly more label information than our method.

The third baseline is HSGG's geometry-aware relation filter, reimplemented on top of the same scorer. It represents the hand-crafted pruning route.

An oracle threshold chosen per scene with target labels gives the ceiling.

Finally, an ablation isolates the key ingredient. It uses shadow pairs that are built naively: uniform random object pairs with no matching of the host scene's geometry statistics. If this version also controls FDR, the careful construction is unnecessary. If it fails, matched shadows are the contribution.

All methods share the scorer, candidate set, and compute. The only tuned quantity, $q$ or K, is swept identically for everyone.

For context on scorer quality, the file reports SceneGraphFusion's Relationship Recall@50 of 0.85 on 3DSSG and OCRL-3DSSG's relationship R@1 of 91.27%. These are recall numbers at fixed K. For emitted-edge FDR on 3DSSG, and for conformal calibration of scene graphs, no reported result was found on Lacuna.

Beating the baselines means two things. First, realized FDR must stay within ±0.02 of $q$ across target domains, in cases where the conformal threshold drifts beyond that. Second, at matched realized FDR, recall must be higher than top-K in scenes whose object counts differ from the training average.

Why not just…?

Why not just calibrate one threshold with conformal prediction on labeled validation data? In-domain this is a real threat, and we expect it to match us there. Our advantage exists only if the calibration data stop resembling the test scenes: a new dataset, new categories, or a very different scene size. The transfer experiments below are designed to settle this. If conformal holds FDR under those shifts, the proposal reduces to a label-free convenience, and we will report it that way.

Why not just keep top-K and pick K well? A fixed K ignores that the number of true relations scales with scene content. GroupForward's reliance on a fixed candidate bundle shows the failure mode: omitted items are unrecoverable. We measure recall at matched precision, broken down by object count, to show where fixed K loses.

Why not just prune by geometry or improve the scorer? Geometric filters like HSGG's still leave hallucinated predicates that HSGG itself had to remove with contrastive decoding. Better scorers like OCRL-3DSSG or LEO sharpen the scores but still need a cutoff. Our method sits on top of either. We therefore apply it to two scorers of different quality, to show that control holds with both and that the recall gain shrinks, but does not vanish, as the scorer improves.

Proposed Experiments

The in-domain test uses 3DSSG. We train a SceneGraphFusion-style scorer, sweep $q$ from 0.05 to 0.3, and plot realized FDR against target alongside recall. All baselines use the same splits.

The shift test is the decisive one. We score VG150 with a training-free vision-language relation scorer in the HSGG style, calibrating conformal on one category subset and testing on held-out categories. In 3D, we train on 3DSSG and emit graphs for ScanNet scenes, where no relation labels exist. There the graphs feed a downstream grounding step on Sr3D+ in the GaussianGraph pipeline, which reports 18.2% grounding accuracy against 13.3% for ConceptGraph. The prediction is that FDR-controlled graphs improve grounding accuracy over top-K and conformal graphs, because fewer hallucinated edges mislead the reasoner.

The decisive checks are three. Shadow and true-null score distributions must be statistically indistinguishable after matching; if they are not, the method's guarantee is void, and the entropy-stratified test will show where it breaks. Realized FDR must track $q$ under shift where the conformal threshold does not. The naive-shadow ablation must fail, showing that matched construction is what makes the probes valid.