Graded Privilege Masks: One Transformer as Teacher, Student, and Everything Between for Sim-to-Real Manipulation

A common way to train robot policies in simulation is to first train a "teacher" that sees information only the simulator knows, such as exact object poses, masses, friction, and contact states. A separate "student" is then trained to copy the teacher using only camera images and joint readings, since that is all a real robot has. This works poorly when the teacher relies on facts the student cannot infer from pixels: the student is asked to copy behavior it has no way to reproduce. We propose to put teacher and student inside a single Transformer and choose which one is running by changing the attention mask, the matrix that decides which tokens may read which other tokens. The mask can also do more than switch between two roles. It can hide any subset of the privileged tokens, which gives a whole family of intermediate teachers, each seeing a little less of the simulator's secrets. The student then learns from teachers whose knowledge it can actually approach.

The same masking rule handles the uneven data that real sim-to-real co-training produces. Thousands of simulated demonstrations carry privileged state. The few real demonstrations carry none, so they train only the positions that never see privileged tokens. No second network, alignment loss, or domain classifier is needed. The claim is that randomizing over subsets of privileged information through the mask, inside one shared-weight model, closes more of the gap between simulated and real success than either a single hard teacher/student split or external distribution alignment, and that the gain is largest on tasks where some privileged quantity cannot be seen in the image.

Where this sits

The closest work is Unified Locomotion Transformer with Simultaneous Sim-to-Real Transfer for Quadrupeds (ULT). ULT appends privileged physics tokens to the end of a proprioceptive history and uses an ordinary causal mask, so earlier "student" positions cannot see them and the final "teacher" position can. It trains both roles at once with reinforcement learning, online imitation, and dynamics losses. ULT supplies our core mechanism. We differ from it in three ways. ULT has one binary split, and we sample a distribution over privileged subsets. ULT works on low-dimensional proprioception, and we work on image-and-language manipulation. ULT does not co-train on real data, and we do.

Three lines of multimodal work show that one backbone can take on several roles by switching masks: Unified Language Model Pre-training for Natural Language Understanding and Generation for text, UniXcoder for code with prefix-token switches, and UP-VLA for robots, where a flexible mask alternates between question answering and future-frame prediction. These papers motivate the design. Our contribution borrows their "mixture of masks during training" recipe and applies it to privilege rather than task type, which none of them do. Unified Vision-Language-Action Model (UniVLA) uses plain autoregressive masking with no privileged channel, so it serves as a strong external reference on the benchmarks rather than a mechanism we reuse.

Two papers constrain the design. Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training finds that naively pooling many simulated and few real trajectories fails because the policy solves each domain separately. The authors fix this with an external unbalanced optimal transport loss, a method that matches distributions while allowing their total mass to differ. This finding is a live risk for us: a mask that treats domains differently could simply help the model keep them apart. Adversarially Masking Synthetic to Mimic Real shows that domain discriminators latch onto trivial "clean versus noisy" cues. This is why we avoid a discriminator entirely. Mask-Based Priors Are More Persistent than Query-Key Initializations shows that structure placed in the attention mask survives training, whereas structure placed in initial weights washes out. This supports putting the privilege structure in the mask. Here the structure is enforced exactly by construction, so we do not need that paper's learned soft mask.

Core Claim

Consider a sequence made of visual tokens, a language instruction, proprioceptive and action history, and $K$ privileged tokens $p_1,\dots,p_K$. For a subset $S \subseteq \{1,\dots,K\}$, define a mask $M_S$. Under $M_S$, a readout position tagged with $S$ may attend to all observation tokens and to $p_k$ only for $k \in S$. Observation tokens never attend to any privileged token, so real-robot inference with $S=\emptyset$ is an exact special case with no leakage. All subset-tagged readouts share one set of weights and are computed in one forward pass. Training minimizes

$$\mathcal{L} = \mathbb{E}_{S \sim q}\big[\ell(\pi_S(a \mid o, p_S), a^\star)\big] + \lambda\, \mathbb{E}_{S \sim q}\big[\mathrm{KL}\big(\pi_{S}(\cdot \mid o, p_S)\,\|\,\pi_{S'}(\cdot \mid o, p_{S'})\big)\big],$$

The symbols are as follows. $o$ is the non-privileged observation. $a^\star$ is the demonstrated action. $\ell$ is the behavior-cloning loss. $q$ is the distribution over privileged subsets. $S' \subset S$ is a subset obtained by removing one element from $S$. The KL term, whose gradient flows only into the less-privileged branch $\pi_{S'}$, makes each teacher distill into a slightly weaker one, forming a chain that ends at the student $\pi_\emptyset$. Real demonstrations contribute only the $S=\emptyset$ term.

The prediction is specific. The benefit over a single binary split should grow with the number of privileged quantities that are not identifiable from vision, such as mass or friction. For quantities like object pose, which the camera largely reveals, the graded chain should add little.

Research Program

We build the model at two scales. The first is a mid-size Transformer policy trained from scratch, which allows cheap ablations. The second is a fine-tuned UP-VLA-style unified backbone, which tests whether the result holds on a pretrained multimodal model.

In simulation, we run ULT's action-mixer scheme. We roll out a mixture of the student and a sampled graded teacher, and relabel the visited states with the full teacher's action, so the student learns to recover from its own mistakes.

We then measure the "imitation gap" directly. For each privileged subset, we record the action disagreement between $\pi_S$ and $\pi_\emptyset$, and we check whether the chain reduces the student's disagreement with the teacher it is learning from.

We then carry the trained student to real hardware with a small set of real demonstrations. The question is whether the $S=\emptyset$-only rule for real data avoids the domain-separation failure that Cheng et al. report. To test this, we probe whether a linear classifier can tell sim from real in the student's hidden states, and whether target-region skills learned only in simulation transfer.

Baselines

The naive reference point is a single student-only policy trained by behavior cloning on pooled simulated and real demonstrations, with no privileged input at all. It uses the same architecture, data, and training steps. Beating it is the minimum bar.

The strongest privileged-information competitor is standard two-stage distillation. A full-privilege teacher is trained first, and a separate student network is then trained with on-policy relabeling in simulation. We give it the same total simulator step budget as our single model, including the teacher's steps, and the same hyperparameter search budget of equal trial counts. Winning requires higher real and held-out success at equal budget, not just equal success at half the budget. The latter would replicate ULT's efficiency claim, which is not our claim.

The strongest co-training competitor is the unbalanced optimal transport method of Generalizable Domain Adaptation for Sim-and-Real Policy Co-Training. We run it on our six sim/real task pairs with its reported regime of 200 to 1,000 simulated and 10 to 25 real trajectories, using the same backbone. The file reports it gives up to a 30% improvement in real success over standard co-training, but gives no absolute per-task numbers. We will therefore reproduce it rather than quote it, and beating it means higher mean real success across tasks, measured with matched evaluation seeds.

On CALVIN ABC→D, the external reference is UP-VLA at an average completion length of 4.08 (GR-1 at 3.06). On LIBERO it is UniVLA at 95.5% average success, and on SimplerEnv-Bridge UniVLA reports 69.8%. For our fine-tuned-backbone variant, the fair comparison is the same backbone fine-tuned without privileged tokens. The absolute numbers indicate whether we are in a competitive range.

The key ablation is ULT's binary mask: only $S=\emptyset$ and $S=$ full, with identical everything else. This isolates graded subsets as the new ingredient. A second ablation keeps graded subsets but gives each role separate weights, which isolates weight sharing.

Why not just…?

Why not just add an auxiliary head that predicts the privileged state from images? This is the most serious threat, because it also injects simulator knowledge into the shared representation and is simpler. However, it forces the network to predict mass or friction even when those quantities are not visible, which can teach it to hallucinate them from rendering cues, the same shortcut behavior described for simulator artifacts. We run it as a baseline with a matched budget. If it matches graded masks on the non-identifiable-privilege tasks, we will report that the mask adds nothing beyond auxiliary supervision.

Why not just use two-stage distillation with separate networks? ULT reports that separate distillation costs tens of millions of extra simulation steps for Transformer policies. More importantly, a single full-privilege teacher leaves the imitation gap untouched. The separate-weights graded ablation and the matched-budget two-stage baseline together show whether sharing weights or grading privilege accounts for any gain.

Why not just use domain randomization? Randomization widens the range of simulated appearance and physics. It does not tell the student which behaviors depend on facts it cannot see. It is also complementary: every method in our comparison, including ours, is trained on randomized simulation, so randomization is held constant rather than competing.

Proposed Experiments

We start with CALVIN ABC→D and LIBERO. Both simulators expose object and articulation state that can serve as privileged tokens, and CALVIN's unseen environment D tests transfer to a new layout. We add a controlled suite in which we vary object mass and surface friction, which are invisible to the camera, alongside object pose, which is visible. This gives a curve of the graded-versus-binary advantage against the number of non-identifiable privileged factors, and that curve is the decisive test of the core claim.

For real transfer, we build six Franka tasks paired with simulated counterparts, following Cheng et al.'s data regime, and include target regions covered only in simulation. We report real success over matched trials, the sim-to-real success drop, and the linear-probe accuracy for telling sim from real in student hidden states. SimplerEnv-Bridge serves as a secondary real-to-sim check.

The claim is falsified if either of two results holds. The first is that graded masks do not beat the binary mask on the non-identifiable-privilege tasks. The second is that the $S=\emptyset$-only real-data rule yields probe accuracy near 100% together with no transfer to target regions seen only in simulation. That would mean the mask became the domain-separating bypass that Cheng et al. warn against.