Committing to a Mode: Hierarchical Distillation of Domain-Randomized Teachers for Sim-to-Real Transfer

A common way to get a robot policy that works on real hardware is to train many specialist policies in simulation, each on a narrow slice of randomized physics such as low friction or a heavy object, and then compress them into a single deployable policy. Compression usually means regression: the student is trained to output the average of what the teachers would do. This proposal argues that the step quietly breaks exactly where it matters. When the specialists disagree, for example one pushes from the left because the box sticks and another pushes from the right because it slides, the regressed student outputs something in between that neither specialist would ever do. The student averages most in the first moments of an episode, before its sensor history has revealed which physics it is in.

Our core claim is that this "behavioral averaging" during distillation is a partial-observability problem with a hierarchical fix. The student should explicitly choose which teacher mode to follow. It should hold that choice for a stretch of time, and keep the choice soft, weighted by how good each teacher's own value estimates say that mode is, until the history has disambiguated. Simply swapping in a more expressive student, such as a diffusion model that can sample any mode, is not enough. Such a model resamples its mode at every control step, and hopping between incompatible strategies averages them over time instead of within one action. We will build this two-level student, measure averaging and mode-hopping directly, and test it on zero-shot transfer to a physical Franka arm.

Where this sits

The teacher stage comes from Cyclic Policy Distillation (CPD). CPD splits a wide physics randomization range into overlapping sub-domains, trains a local policy on each with Soft Actor-Critic (a maximum-entropy RL method that rewards the policy for staying random while earning reward), and distills them into one global policy. CPD reports 3x to 6x better sample efficiency than global randomization on Pendulum. It also reports that plain randomized SAC failed to learn HalfCheetah at all. Its distillation step is unimodal regression, so CPD motivates our work by showing that local teachers are worth having, and it supplies the teachers. The averaging diagnosis comes from imitation learning. Regression onto conflicting demonstrations returns their mean, which CE-Nav calls "disastrous averaging" and avoids with a normalizing-flow planner. Expressive students such as Diffusion Policy and VQ-BeT avoid averaging within a single action. VQ-BeT itself reports jerky transitions at token boundaries, which is a hint of the cross-step problem we target. X-Sim already distills simulated RL rollouts into a diffusion policy, but it uses one teacher, not a family of physics-specialized teachers whose modes conflict.

On the sim-to-real side, DORAEMON avoids distillation altogether. It trains one history-conditioned SAC policy while maximizing the entropy of the physics distribution, subject to a success-rate constraint. Privileged Knowledge Distillation (HIB) infers hidden physics from proprioceptive history inside a single RL loop. Both are strong competitors because they bet that history-conditioning makes the problem unimodal, and our proposal must beat that bet. One result constrains our design. KL-Regularized RL is Designed to Mode Collapse shows that flexible policies trained under weak KL regularization collapse onto one mode. If our mode-selector were trained with a hard mode-seeking objective, it would commit too early and fail whenever the guess is wrong. This is why the selector target below is a soft Boltzmann distribution. The hierarchical framing borrows from DIPPER, which uses the low-level worker's value to keep a manager's choices feasible. We apply the analogous idea with teacher Q-values scoring which modes are viable, but our high level picks among behavioral modes rather than spatial subgoals.

Core Claim and Method

We start with $K$ CPD teachers $\pi_k$, each with a soft Q-function $Q_k$ trained in sub-domain $k$. The student has two levels. A selector $q_\phi(z \mid h_t)$ reads the observation-action history $h_t$ and chooses a teacher index $z$. The chosen $z$ is held for $T$ steps, or until the selector's distribution shifts by more than a threshold. A low-level policy $\pi_\theta(a \mid s_t, h_t, z)$ is a fast unimodal Gaussian, because within one mode the action distribution really is unimodal.

The selector is trained in simulation, where the true physics parameters $\xi$ are known. Its target is a soft assignment over teachers:

$$ p^\star(z = k \mid s_t, \xi) \propto \exp\big(Q_k^{\xi}(s_t, \pi_k(s_t)) / \alpha\big) $$

Here $Q_k^{\xi}$ is teacher $k$'s soft value, re-estimated by short rollouts of teacher $k$ under the true parameters $\xi$, and $\alpha$ is a temperature. The selector minimizes cross-entropy to $p^\star$ given only the history. Two consequences follow. When several teachers succeed under $\xi$, the target spreads mass across them, which is the maximum-entropy principle applied at the level of modes. When the history cannot yet distinguish domains, the selector's best fit is the average of these soft targets over plausible $\xi$. That average is a mixture over discrete modes, and sampling from it commits to one coherent teacher rather than to their arithmetic mean.

The low-level policy is trained by DAgger-style relabeling: the student acts, and the selected teacher labels the action. Rollouts are drawn over the full randomization range. The design yields two measurable quantities. The first is an averaging index: at each state, the distance from the student's action to the nearest teacher action, divided by the distance to the mean teacher action. The second is a mode-switch rate, the number of changes per episode in which teacher the student's action is closest to.

Research Program

The first phase is diagnostic. We build a PandaPush variant with a deliberately bimodal friction regime, so that sticking and sliding call for opposite push sides. We measure the averaging index and mode-switch rate over the first $N$ steps of an episode for every student type. We also measure the identification window, defined as the step at which a history-conditioned probe can classify the sub-domain with 90% accuracy. The prediction is that averaging is concentrated inside this window. If it is not, the mechanism is wrong and we will say so.

The second phase compares students on the standard 17-parameter PandaPush randomization from DORAEMON and on randomized HalfCheetah from CPD. All students share the same teachers. The third phase deploys zero-shot on a physical Franka Panda pushing a box with unknown center of mass, using DORAEMON's protocol: success within 3 cm, plus final distance.

Baselines

The naive reference point is CPD's own distillation: an MLP student trained by mean-squared-error regression on the same teachers and the same rollout states. It is the method our diagnosis says fails. Beating it on the bimodal variant is necessary but not sufficient.

The strongest existing method is DORAEMON, a single history-conditioned SAC policy trained with entropy-maximizing randomization. On physical PandaPush it reports 60% zero-shot success with an average final distance of 2.68 cm, against 46.7% for LSDR, 26.7% for AutoDR and 0% for fixed randomization (DORAEMON). We will rerun DORAEMON with its released settings, using the same total simulator steps as our teachers plus distillation combined, and the same hyperparameter-search budget. Beating it means higher physical success over at least 30 trials with non-overlapping 95% intervals, or equal success at lower final distance. In simulation, we must win on the held-out extremes of the parameter range.

The history-conditioned regression student follows HIB: a temporal-convolution history encoder with an MSE head, trained on the same teachers. This tests whether information alone removes averaging. The expressive-student baseline is a history-conditioned diffusion student in the style of Diffusion Policy, with a normalizing-flow variant following CE-Nav to remove the latency confound. It resamples at every step. We also include an action-chunking student in the style of ACT, with and without temporal ensembling. Temporal ensembling explicitly averages overlapping chunks, which makes it an interesting negative control.

The ablations isolate the key ingredients. Setting the hold length $T=1$ turns our model into a per-step mixture, which isolates temporal commitment. Replacing $p^\star$ with a one-hot argmax target isolates the soft maximum-entropy assignment, and we predict premature commitment as the KL mode-collapse analysis implies. Replacing the history-based selector with an oracle given the true sub-domain gives an upper bound.

Why not just…?

Why not just condition the regression student on history? This is the most serious threat. If the history identifies the physics quickly, the conditional action distribution becomes unimodal and plain regression stops averaging. HIB's 100% success over 10 trials on real 0.6 m stairs (HIB) shows how far history inference can go. Our diagnostic phase settles this directly. If the identification window is only a few steps and the HIB-style student matches ours, the hierarchy is unnecessary on these tasks, and we will report that as the finding. We expect a gap on contact tasks, where friction and mass are only revealed after contact, which is precisely when committing to a wrong push side is costly.

Why not just distill into a diffusion or flow student? These models do not average within one action, but nothing ties the mode chosen at step $t$ to the mode chosen at step $t+1$. The $T=1$ ablation and the mode-switch rate test whether per-step resampling produces hopping that degrades success. Diffusion also costs 573 ms per action on hardware against 18 ms for VQ-BeT (VQ-BeT), which matters for closed-loop pushing. The flow baseline removes this latency excuse, so any remaining gap must come from commitment.

Why not just skip distillation and train one policy, as DORAEMON does? DORAEMON is a real competitor and may simply win on PandaPush. However, CPD reports that single-policy randomized SAC failed outright on wide HalfCheetah ranges. DORAEMON also notes forgetting when it backtracks to narrower distributions. The two-level student targets the regime of wide ranges where conflicting modes appear, and the HalfCheetah and bimodal-friction comparisons will show whether that regime exists in practice.

Why not just use action chunking for commitment? Chunks commit, but they do so open-loop and at a fixed length. ACT's temporal ensembling smooths chunks by averaging them, which reintroduces the problem. The chunking baselines measure whether fixed open-loop commitment is enough or whether history-triggered re-selection is needed.

Decisive Checks

Three outcomes would decide the question. First, on the bimodal friction variant, the averaging index for MSE students should peak inside the identification window, while ours stays near teacher modes. Second, the $T=1$ and hard-argmax ablations should each lose success, one through mode hopping and the other through premature commitment. If neither loses, the two components are not doing work. Third, on the physical Franka, the student must exceed DORAEMON's 60% or match it with lower final distance. Inference must stay under 20 ms per step. If the history-conditioned regression student matches us on hardware, we will conclude that sim-to-real distillation averaging is resolved by information rather than by structure. That is still a useful and publishable answer.