Where the Sim-to-Real Gap Lives: Adapting Only the Feedback Pathway of a Simulated Policy

Robot policies trained in simulation usually fail at the moment of contact. Before contact, the arm reaches and positions itself well. When a peg meets a hole or a nut meets a post, the simulator's simplified friction and contact models stop matching reality, and the policy makes mistakes. The same pattern appears in human motor control. Imagined and executed movements share the same motor-cortex activity while the movement is being planned, and they diverge only after movement begins, when real touch and joint-position feedback start arriving. Common neural correlates of real and imagined movements showed this with brain recordings (MEG), where decoding patterns for real and imagined grasps overlapped. We propose to treat a simulated rollout the same way, as motor imagery. The planning part of a sim-trained policy should transfer largely intact. The gap should be concentrated in the part of the network that reacts to feedback after contact.

This gives a claim that can be tested and then used. First, we measure where inside a policy, and when during an episode, simulated and real internal activity diverge. If the divergence is concentrated in the feedback-conditioned layers after contact, then a small real-world adaptation budget should be spent only there. Doing so should match or beat fine-tuning the whole network, with fewer real episodes and without damaging the approach behavior that already works.

Core Claim

Our policy has two pathways, following the design of DECO. A planning pathway turns camera images and the goal into chunks of future actions. A feedback pathway injects joint positions, wrist force-torque, and optionally touch through lightweight conditioning layers. DECO feeds proprioception in through adaptive layer normalization, which rescales intermediate features based on the robot's joint state. It feeds touch in through a plug-in cross-attention adapter. We hypothesize two things. After training in simulation, the sim-versus-real divergence in hidden activity will be small during the approach phase and will grow sharply after first contact, mostly inside the feedback pathway. Adapting only the feedback pathway on a few real episodes will then close most of the gap.

DECO itself trained entirely on 50 hours of real teleoperation and never tested sim-to-real transfer. We keep its architecture and change where the data comes from and which parameters are adapted. The new part is the pairing of a phase-resolved divergence diagnostic with an adaptation rule chosen by that diagnostic. Without the diagnostic, this would reduce to "LoRA on some layers," where LoRA is low-rank parameter-efficient fine-tuning.

Research Program

Training in simulation. We train the two-pathway policy in robosuite on a Franka Panda. Actions are operational-space (Cartesian) commands, which that paper found learn faster and transfer more easily than joint-velocity control. Tasks are Nut Assembly, Pick-and-Place, and a peg insertion task; robosuite reports Nut Assembly and Pick-and-Place as unsolved under reinforcement learning from scratch. To get there, we generate demonstrations with a reinforcement learning teacher that sees privileged object state, following Asymmetric Actor Critic. We then distill that teacher into the image-and-proprioception student. Dynamics randomization follows Peng et al.: we randomize masses, damping, friction, controller gains, and latency.

Diagnosis. We run the sim-trained policy on the real robot and in a reconstructed simulated copy of the same scene, from matched initial configurations. For each layer and each timestep, aligned to first contact detected by force-torque, we compute two measures. The first is how well a linear classifier separates sim activity from real activity. The second is how well a linear probe decodes the upcoming action, which mirrors Sugata et al.'s time-resolved decoding of real versus imagined movement. The premise is supported if three things hold together: probe accuracy stays high and sim/real separability stays low before contact in the planning pathway, and separability rises after contact mainly in feedback layers.

Adaptation. Using K real episodes, with K ∈ {5, 10, 20, 50} demonstrations or corrective interventions, we update only the feedback pathway, meaning the conditioning and cross-attention parameters. The planning pathway stays frozen. STEADY plays a related but different role. It keeps simulator skills fixed and learns real-world skills constrained to be orthogonal to them, which reduced quadrotor tracking error by 30.2% versus simulator zero-shot. STEADY assumes the dynamics decompose linearly and was tested only on flight. We borrow its principle of keeping transferred structure fixed and learning only what is missing. We apply it through the network's architecture rather than through a spectral decomposition, and we target contact-rich manipulation, where that linearity assumption is least plausible.

A cheaper "pseudo-real" testbed lets us run many seeds before touching hardware. It is robosuite with held-out contact parameters (soft-contact stiffness, friction far outside the randomization range) plus unmodeled backlash and latency. Physical Panda experiments are the final test.

Baselines

Zero-shot transfer. The naive reference point is the domain-randomized sim policy deployed with no adaptation, the recipe of Peng et al. The number to beat for randomization is theirs: a 0.89 ± 0.06 real success rate on Fetch puck pushing with a recurrent policy, versus 0.67 for a feedforward policy with randomization. That task is single-contact planar pushing, so we expect our zero-shot numbers on insertion to be much lower. We found no reported result on Lacuna for sim-to-real Nut Assembly or peg insertion on a Panda.

Full fine-tuning. The strongest competitor is fine-tuning every parameter on the same K real episodes. It has the most capacity to fix the gap, so our method must at least match it. We will match the number of gradient steps, give each method the same hyperparameter-search budget (learning rate and number of epochs, 12 configurations each), and use identical real data. Beating it means equal or higher final success at K ≤ 20, with approach-phase success that does not drop below the zero-shot level.

Real-only training. A DECO-style policy trained from scratch on the K real episodes tests whether simulation contributes anything at all. For reference, DECO reports 72.25% vision-only and 82.50% with its touch adapter on DECO-50, after 50 hours of real data. That is not directly comparable to our setting, but it bounds what real-only training achieves when data is abundant.

Two adapter controls. These isolate the key ingredient. The first adapts LoRA across all layers with the same parameter count as our feedback-only adapter. The second, the inverse ablation, adapts only the planning pathway. If our hypothesis holds, the planning-only adapter should help little, and the all-layer adapter should overfit more at small K.

Model-side correction. We include a neural-augmented simulator in the style of NAS, which learns simulator-versus-real residual dynamics with an LSTM and then retrains the policy in the corrected simulator. It gets the same real transitions.

Why not just…?

Why not just fine-tune everything on the real episodes? With 5 to 20 episodes, updating the whole network risks overwriting approach behavior that already transfers. The diagnostic predicts exactly this: the approach phase is where sim and real already agree. The comparison is direct. We report success broken down by phase, and we test whether full fine-tuning lowers approach success relative to zero-shot. If full fine-tuning matches us on both the approach phase and completion, the adaptation rule adds nothing, and the contribution shrinks to the diagnostic.

Why not just put generic LoRA everywhere? This is the most serious threat, because parameter-efficient tuning is often reasonably robust. The budget-matched all-layer LoRA control exists to settle it. If all-layer LoRA does as well as feedback-only adaptation, we will report that the location of adaptation does not matter at this scale. The phase-localization finding would still stand as a scientific result about transfer. What would favor our design is an advantage that grows as K shrinks, together with the inverse ablation clearly underperforming.

Why not just randomize contact parameters more widely? Wide randomization makes policies conservative, and it cannot represent effects such as backlash or friction hysteresis that the simulator lacks entirely. NAS was motivated by exactly this. Our pseudo-real testbed includes an arm with twice the randomization range, which directly tests whether breadth alone closes the post-contact gap.

Why not just correct the simulator, as NAS does? Correcting the simulator requires real exploration data that covers the deployment states, plus matched sim-real state pairs. Insertion near contact is exactly where exploration risks damaging the hardware. We run NAS with the same data, and it may well win on smooth-dynamics tasks. We expect it to struggle where real contact states are sparse in the collected data.

Proposed Experiments and Decisive Checks

We run three tasks in the pseudo-real testbed (five seeds each) and three tasks on a physical Panda (30 trials per condition). We report overall success, approach-phase success, completion given contact, and real episodes needed to reach 80% of full-fine-tuning performance.

The first decisive check is the diagnostic. If sim/real separability before contact is as high as after contact, or if divergence is spread evenly across both pathways, the imagery analogy fails and we stop before building the adaptation method.

The second check is the inverse ablation. Adapting only the planning pathway must recover clearly less than adapting only the feedback pathway at every K.

The third check is data efficiency. Feedback-only adaptation should match full fine-tuning at K ≤ 20 without lowering approach-phase success. If it does, a practical rule follows: spend real robot time only on the part of the controller that reacts to contact.