Simulators know things that real robots and cameras never will: which object hides which, how much each link of a robot body weighs, which surfaces touch. A common trick for sim-to-real transfer is to train a "teacher" model that sees this privileged information in simulation, then train a deployable "student" that sees only real sensor data to imitate the teacher. The trouble is that the student imitates everything, including relations that exist only because the simulator is idealized and that no real sensor could ever reveal. We propose to represent the privileged information as a graph, with objects or body parts as nodes and their relations as edges. The teacher then learns, edge by edge, which relations to pass on. It keeps a relation only if the relation helps the task and the student can actually predict it from what it will see in the real world.
The mechanism comes from the graph information bottleneck. This is a family of graph neural networks (GNNs, models that pass messages along the edges of a graph) that put a learned, noisy gate on every edge and pay a penalty for every bit of graph information they let through. That forces them to keep only a minimal subgraph that still predicts the label. Our change is to train these gates jointly with the student and route the student's imitation error back into them. Edges whose information the student cannot reconstruct then become expensive and close. The result is a teacher that hands over a subgraph of transferable relations rather than a full copy of the simulator's world model. It also yields an inspectable list of which simulated relations crossed the domain gap.
Core Claim
The claim is that, for graph-structured privileged information, the bottleneck should sit at the distillation boundary and be shaped by what the student can recover, not only by what the task needs. Let $G^p$ be the privileged simulator graph, $Y$ the task target, $o$ the student's real-world-available observations, $m_e \in [0,1]$ the stochastic gate on edge $e$ with keep-probability $p_e$, $Z_T$ the teacher representation computed from the gated subgraph, and $Z_S = f_S(o)$ the student representation. We train
$$ \mathcal{L} = \mathcal{L}_{task}(Y \mid Z_T) + \beta \sum_{e} \mathrm{KL}\big(\mathrm{Bern}(p_e)\,\|\,\mathrm{Bern}(r)\big) + \lambda \,\| Z_S - Z_T \|^2 , $$
where $r$ is a small prior keep-rate, $\beta$ sets compression strength, and $\lambda$ weights distillation. The edge-gate and KL form comes from Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism. What is new is that the last term's gradient flows into the gates $p_e$ as well as into the student. Standard distillation stops the gradient at the teacher. Here, an edge carrying information that is unpredictable from $o$ raises the distillation loss and gets closed.
The claim implies a regime of benefit, and we state it so it can be falsified. Distillation helps when relations are recoverable from $o$ in principle but hard for the student to discover from task supervision alone. A large $\lambda$ collapses the teacher toward what the student already sees and should erase the gain. A small $\lambda$ reverts to ordinary distillation, which transfers simulator shortcuts. We expect a U-shaped curve in real-domain performance as a function of $\lambda$. Its absence would count against the idea.
Position Relative to Prior Work
The closest work is the Historical Information Bottleneck (HIB), described in Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective and Privileged Knowledge Distillation for Sim-to-Real Policy Generalization. HIB compresses a robot's proprioceptive history into a latent that predicts a flat privileged vector (friction, mass, terrain). It reports 100% success over 10 trials on 0.6 m stairs with a real Unitree A1. Those authors state the method is limited to low-dimensional privileged knowledge, and their bottleneck acts on the student's input history, not on the privileged side. We put the bottleneck on the privileged graph and make it relation-selective, so the teacher decides which structure to transfer. HIB motivates this proposal and is our main control baseline.
Three graph papers supply the method and its limits. Graph Information Bottleneck introduced Bernoulli and categorical edge sampling for compression. Those variants improved accuracy over a graph attention network on Pubmed under adversarial edge attacks by 34.0% and 31.3%. GSAT turned this into continuous edge gating with a KL penalty. Causal Subgraphs and Information Bottlenecks (CSIB) constrains our design in a useful way. It shows that invariance methods fail when invariant features only partly determine the label, and that adding a graph bottleneck repairs this, with a 12.6% relative gain on GOOD-Motif size shift. Simulated privileged features are partially informative in exactly this sense. CSIB also warns that results are sensitive to $\beta$, so we sweep it rather than tune it to one value. None of these three papers involves a teacher, a student, or a physical domain gap.
On the distillation side, Frameless Graph Knowledge Distillation shows GNN teachers can be distilled into plain MLP students that match them with no graph at test time. That supports our premise that a topology-free student can absorb graph knowledge. However, it distills everything and stays on clean citation graphs. An Image-enhanced Molecular Graph Representation Learning Framework distills across modalities into GNNs and reports 73.89% average MoleculeNet ROC-AUC, but with no domain shift. TWIST and Cyclic Policy Distillation distill across sim-to-real with vector states and no structural selection.
Sim2SG shows relational structure helps cross the gap: KITTI "Hard" mAP rises from 0.234 with standard domain randomization to 0.316. It uses adversarial alignment, not distillation, and has no filter on noisy relations.
Research Program
We will build the gated privileged teacher, the student, and the joint training loop, then study it in three settings of increasing realism.
The first setting is a controlled diagnostic built on GOOD-Motif. The teacher sees the full graph. The student sees node features plus a sensor-limited subset of edges. We plant "simulator-only" edges that correlate with the label during training, are absent at test time, and cannot be recovered from the student's view. Ground truth for which edges are motif, recoverable, or simulator-only lets us measure gate selectivity directly, using the interpretation-AUC protocol of GSAT.
The second setting is control on the Privileged DeepMind Control Suite used by HIB. We build the privileged graph from each task's kinematic tree: one node per body, edges for joints and contacts, per-node friction and mass as features. The student is HIB's temporal convolution encoder over proprioceptive history. Evaluation follows the benchmark's ordinary, out-of-distribution, and far out-of-distribution splits for dynamics and morphology shifts.
The third setting is perception on IRAIS. Training uses only the synthetic 3D-IRAIS scenes. The teacher sees the simulator's occlusion-order graph between objects (which object is in front of which) together with object features. The student is an amodal segmentation network that predicts the full extent of partly hidden objects from what the deployed model can see. Testing is on human-annotated Real-IRAIS.
Analysis has two parts. We will record which edge types survive gating at convergence, for example contact edges versus mass-ratio edges, or occlusion edges between adjacent versus distant objects. We will also check whether surviving edges predict real-domain gains when restored or removed one type at a time.
Baselines
The naive reference point is the student trained on simulated data alone with task loss and domain randomization, with no teacher. If our method does not beat this on the real or far-out-of-distribution split, the privileged graph contributes nothing. A second reference is standard two-stage distillation. An unconstrained GNN teacher is trained on the privileged graph, then the student matches its representation by squared error. This is the method that, by our hypothesis, transfers simulator shortcuts. On the diagnostic it should show high reliance on planted edges.
In control, the strongest existing competitor is HIB. We give it the same privileged information flattened into a vector of node and edge features, run with the authors' architecture, and give it the same simulator steps and tuning budget as ours. The file reports no scalar for HIB on Privileged DMC; HIB is described as consistently outperforming RMA, DreamWaQ, and Dropper and nearly matching an oracle. We will rerun it. Beating it means higher return on the far out-of-distribution split across at least five seeds, with non-overlapping confidence intervals.
In perception, the strongest competitor is EUREKA from Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer, which reports 37.10% AP on Real-IRAIS after training only on 3D-IRAIS. Our student uses EUREKA's backbone and exactly the test-time inputs EUREKA uses; we will check whether that includes depth. The only added ingredient is the privileged teacher during training, on the same synthetic data with matched training epochs. Beating it means exceeding 37.10% AP on Real-IRAIS in our own reproduction, measured against our reproduced EUREKA number rather than the reported figure alone. For scene-graph transfer, the comparison number is Sim2SG's 0.316 mAP on KITTI "Hard".
The ablations isolate the key ingredient. The first, "GSAT teacher then distill," trains the gated teacher with task loss only, freezes it, and then distills. It has the same bottleneck and the same $\beta$, but the student's error never reaches the gates. The second replaces edge gates with node-level gates. The third replaces learned gates with random edge dropout at a matched keep-rate. Every variant gets the same sweep over $\beta$ and $\lambda$, and we report the full curve, not the best point.
Why not just…?
Why not just flatten the privileged graph and use HIB? This is the most serious threat. If the relevant structure is small, such as a few links with fixed connectivity, a flattened vector may carry everything an edge-gated teacher would. Flattening loses two things. First, it cannot drop one relation while keeping another that shares a node. Second, it has no fixed size when object counts vary, which rules it out for IRAIS scenes. The Privileged DMC comparison settles the first point on HIB's home ground. If flattened HIB matches us there, we will narrow the claim to variable-size relational settings such as IRAIS, and report that outcome.
Why not just train a bottlenecked GSAT teacher and distill it normally? A task-only bottleneck keeps whatever predicts the label in simulation. CSIB's analysis implies this includes partly informative, simulator-specific signal. Nothing in that objective knows what a real sensor can see. The frozen-GSAT ablation tests this directly. On the diagnostic it should keep planted simulator-only edges that our gates close. If it does not, and real-domain numbers match ours, the recoverability coupling is unnecessary and the contribution reduces to "use GSAT as a teacher."
Why not just widen domain randomization? Randomization does not tell the student which relations matter. Sim2SG reports that standard randomization reaches only 0.356 Recall@20 on CLEVR against 0.888 with structured alignment. HIB reports randomization-based baselines degrading in far out-of-distribution tests. Our naive baseline already includes randomization, so this is measured, not assumed.
Why not just give the student a GNN over a predicted scene graph? This is Sim2SG's route. It requires a reliable graph predictor on real data, and predicted relations are noisy. We run a student that predicts the occlusion graph and reasons over it as an additional IRAIS baseline. Our method should match it at lower test-time cost, or beat it if prediction errors in real scenes dominate.
Decisive Checks
Three results decide the proposal. First, on the diagnostic, the coupled gates should assign low keep-probability to planted simulator-only edges and high keep-probability to recoverable motif edges. The frozen-GSAT teacher should not separate the two. Second, real-domain or far out-of-distribution performance should be U-shaped in $\lambda$, with our best point above both endpoints. Third, our method should beat flattened HIB on Privileged DMC far out-of-distribution, and beat our reproduction of EUREKA, reported at 37.10% AP, on Real-IRAIS. Failing the first check means the mechanism does not do what we claim. Failing the third while passing the first two means it works but is not needed, and we would report that as the finding.
References
- Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism
- Bridging the Sim-to-Real Gap from the Information Bottleneck Perspective
- Graph Information Bottleneck
- Causal Subgraphs and Information Bottlenecks: Redefining OOD Robustness in Graph Neural Networks
- Frameless Graph Knowledge Distillation
- An Image-enhanced Molecular Graph Representation Learning Framework
- TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer
- Cyclic policy distillation: Sample-efficient sim-to-real reinforcement learning with domain randomization
- Sim2SG: Sim-to-Real Scene Graph Generation for Transfer Learning
- Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer