A robot policy trained in a simulator usually fails on hardware because of small errors in the simulated physics. Friction, motor lag and contact stiffness are slightly wrong, and the policy's precise motor commands are tuned to those wrong values. We conjecture that such errors rarely change the coarse decision (push left or right, open or close the gripper, lift high or low). They mostly change the fine correction on top of it. We propose to make this structure explicit in the policy's output. Each continuous action dimension is written as a short binary code: the first bit picks a half of the action range, the next bit picks a half of that half, and so on. A policy that emits actions bit by bit, coarse to fine, can then be inspected and adapted one bit-depth at a time. If the simulator-to-reality mismatch concentrates in the deep bits, only the small heads that emit those bits need retraining on target-domain data. The coarse behaviour learned cheaply in simulation stays frozen.
The idea pulls together two literatures that have not met. Logarithmic encodings, which represent N options with log₂N bits instead of N one-hot slots, are well established for shrinking combinatorial search. They also have a well-known weakness: two neighbouring values can differ in many bits. Sim-to-real work, separately, has shown that mismatch is often localized. The proposal asks whether a logarithmic action code turns "where the mismatch lives" into "which bits are wrong," and it fixes the weakness that would otherwise break that correspondence.
Core Claim
Consider a coarse-to-fine binary action code in which each bit's sub-intervals overlap, so that a target near a midpoint is reachable from either branch. We claim three things about it. First, it concentrates the sim-to-target policy discrepancy in the deepest bits. Second, it therefore allows target-domain adaptation of only the deep-bit heads, at a fraction of the samples needed by fine-tuning a flat categorical actor. Third, it keeps the optimization stability that discretized actors bring.
The policy factorizes as
$$\pi(a\mid h)=\prod_{j=1}^{D}\prod_{\ell=1}^{L}\pi_\ell\big(b_{j,\ell}\mid h,\,b_{j,<\ell}\big),$$
where $h$ is the observation history, $D$ is the number of action dimensions, $L=\lceil\log_2 K\rceil$ is the code depth for $K$ bins, and $b_{j,\ell}$ is the $\ell$-th bit of dimension $j$. The diagnostic quantity is the per-depth transfer profile
$$\Delta_\ell=\mathbb{E}_{h\sim d_{\text{target}}}\,\mathrm{KL}\big(\pi^{\text{target}}_\ell(\cdot\mid h,b_{<\ell})\,\|\,\pi^{\text{sim}}_\ell(\cdot\mid h,b_{<\ell})\big).$$
It measures how much the bit-$\ell$ decision of a policy trained directly on the target domain differs from the simulator-trained one. The hypothesis is that $\Delta_\ell$ grows with $\ell$. It can be wrong. If $\Delta_1$ is as large as $\Delta_L$, the premise fails, and that is itself informative about the kind of gap present.
Relation to Prior Work
The method builds on discretized actors. RN-D replaces Gaussian policies with K=41-bin categorical heads per dimension. It shows bounded gradient variance and faster, more reliable learning on MuJoCo and ManiSkill, including StackCube. It evaluates only in simulation and uses flat 1-of-K logits, so it offers no handle on which part of an action decision is wrong after a domain shift. We keep its cross-entropy training and regularized residual network and change only the output code.
Logarithmic action decompositions supply the coding idea but leave the open problem we target.
- Fast RL with Large Action Sets using ECOC assigns actions binary code-words and trains per-bit classifiers or sub-MDPs. It reports large speedups (12.5× overall on 100-action Mountain Car) but uses semantically arbitrary codes in toy domains. - Exact Reduction of Huge Action Spaces sequentializes actions into $\lceil\log_2|A|\rceil$ binary steps in environment time. It proves a double-exponential bound improvement, at the cost of a longer effective horizon.
Both papers name "informed codes," where nearby actions share bits, as unsolved. We differ in two ways. Our code is ordinal and coarse-to-fine by construction rather than arbitrary. Our bits are emitted autoregressively within a single control step, so the planning horizon does not grow.
The Hamming-discontinuity problem constrains the design. In a plain binary tree, a target just past the midpoint of the range has a different top bit from one just before it. A small physics shift near such a boundary would then flip the coarsest bit, which breaks the claim that shifts live in deep bits. Overlapping child intervals are our answer: each child covers, say, 60% of its parent. An ablation with non-overlapping natural binary and with Gray coding tests whether the overlap matters.
On the sim-to-real side, several papers motivate or constrain the approach.
- Provably Safe Sim-to-Real Transfer shows that real-sample cost can scale with a localized mismatch region rather than the whole state-action space. Its method is tabular, demonstrated on a 5×5 gridworld. We pursue the same "pay only for the mismatch" principle, but localize along action-code depth, which works with function approximation.
- Understanding Domain Randomization proves that zero-shot transfer needs history-dependent policies, and that its bounds assume the real world lies inside the randomized family. This constrains our design in two ways. We condition on history $h$. We also treat out-of-support shifts as the regime where cheap targeted adaptation, rather than pure randomization, is needed.
- Grounded Action Transformation corrects the simulator using real data through learned forward and inverse models, demonstrated on a NAO humanoid. It is the main alternative way to spend real samples, and we compare against it rather than build on it.
Research Program
The first step is the actor. For each action dimension with K=64 bins (L=6), a shared trunk produces a context. Six small bit heads then emit bits in order, each conditioned on the previous bits through a learned embedding of the interval chosen so far. Training uses PPO with RN-D's weighted cross-entropy and regularized residual network, on domain-randomized simulators.
The second step is measurement. We train a reference policy directly on each target domain and estimate $\Delta_\ell$ for every depth on target-state distributions. We report these profiles separately for dynamics shifts (friction, mass, actuator gain, latency), action-channel noise and observation noise. Our expectation is that dynamics and actuation shifts produce depth-increasing profiles. Observation shifts may not, because they corrupt the context and so all bits. That would mark a boundary of the method rather than a failure.
The third step is adaptation. Using a fixed budget of target-domain transitions, we fine-tune only bit heads at depth ≥ $\ell^\*$, for every cut-point $\ell^\*$, with the trunk frozen. This traces a curve of recovered return against target samples. A cheap rule picks $\ell^\*$ from a small target batch: retrain the heads whose estimated per-bit advantage disagreement is largest.
Baselines
The naive reference point is zero-shot transfer of a domain-randomized Gaussian PPO policy with recurrent memory, as the domain randomization theory requires, evaluated on the target domain without adaptation. It shows how large the gap is before any method acts.
The strongest existing competitor is RN-D, run with its released code alwaysbyx/RND-RL. It is the strongest because it already holds the stability advantage our method inherits. Any gain we show must therefore come from the code structure, not from discretization. We run it zero-shot, then with full fine-tuning and with last-layer fine-tuning, on exactly the same target-sample budgets, randomization ranges, network width and hyperparameter search budget as our method. RN-D reports reaching peak baseline performance 1.3× to 1.9× faster and is the only method that reliably solves StackCube. No reported result was found on Lacuna for RN-D, or any discretized actor, under dynamics shift or target-domain adaptation. We will therefore establish that number ourselves. Beating it means one of two things: matching RN-D's fine-tuned target return with at most half the target transitions, or exceeding it at equal budget, across at least three seeds per task on a majority of tasks.
The simulator-grounding competitor is GAT/SGAT, given the same target-transition budget to fit its forward and inverse models. On hardware it reported a 43.27% walking-speed increase to 27.97 cm/s and 0% falls for SGAT on bumpy carpet. Those are not numbers on our benchmarks, so we compare only on matched sim-to-sim shifts.
Ablations isolate the key ingredients. The first replaces the ordinal code with a random ECOC code of equal length, in the spirit of ECOC factorization. This destroys locality and should flatten $\Delta_\ell$. The second removes interval overlap and tests the boundary-leakage argument. The third varies the branching factor (2, 4, 8, flat 64) to ask whether depth itself matters.
Why not just…?
Why not just use wider domain randomization? The domain randomization theory guarantees transfer only when the target lies inside the randomized family. Robust Gymnasium reports that PPO and SAC degrade sharply under frequent low-intensity perturbations. Randomization is also our training regime, not a rival. The question is what to do once it falls short. Our zero-shot rows show directly how much is left over.
Why not just fine-tune RN-D's flat categorical head? This is a serious alternative. With D×41 logits, it already avoids the $K^D$ blow-up, so the compression argument from the logarithmic-encoding literature does not apply here. Our claim rests only on sample efficiency of targeted adaptation. If last-layer fine-tuning of flat RN-D matches deep-bit-only adaptation at equal budgets, the proposal's central claim is refuted, and the experiment is designed to show exactly that.
Why not just use a two-level coarse/fine categorical head instead of a binary tree? This is probably the strongest threat. It may capture most of the benefit, since the essential ingredient may be coarse-to-fine factorization, not base 2. The branching-factor ablation settles it. If base 8 with depth 2 performs as well as base 2 with depth 6, we will report that the useful object is the hierarchy and that binary depth adds nothing. The log-depth code then reduces to one point on a design curve rather than the contribution.
Why not just ground the simulator with GAT and retrain? Grounding spends target data on learning full forward and inverse dynamics, which can compound errors over long rollouts. Our method spends the same data on a few small heads. Matched-budget comparisons show which use of target data recovers more return. Grounding should win when the shift changes coarse behaviour, and the $\Delta_\ell$ profile should predict when that happens.
Proposed Experiments
The testbeds come from the file. We use ManiSkill state-based tasks (PickCube, StackCube) and MuJoCo Ant and Humanoid. Targets are created with Robust Gymnasium's dynamics, action and observation disruptions, plus held-out friction, mass, actuator-gain and latency values outside the training randomization range. Isaac Gym provides throughput for computing reference target-trained policies, which are needed to estimate $\Delta_\ell$.
The metrics are three: the $\Delta_\ell$ profile per shift type; target return as a function of target-domain transitions (area under that curve, and samples needed to reach 90% of the target-trained reference); and action jerk under observation noise, which checks whether bit-wise outputs cause actuator chatter.
The first decisive check is whether $\Delta_\ell$ rises with depth for dynamics shifts, and whether this disappears under the random-code ablation. If not, the mechanism is absent. The second is whether deep-bit-only adaptation beats fine-tuned RN-D at matched budgets. The third is whether the advantage survives the base-8 hierarchical ablation. Only if all three hold does the log-depth ordinal code earn its place. A physical-arm deployment would then be the natural next step.
References
- RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning
- Fast Reinforcement Learning with Large Action Sets Using Error-Correcting Output Codes for MDP Factorization
- Exact Reduction of Huge Action Spaces in General Reinforcement Learning
- Provably Safe Sim-to-Real Transfer
- Understanding Domain Randomization for Sim-to-real Transfer
- Grounded action transformation for sim-to-real reinforcement learning
- Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning