Align the Frames, Not the History: Where Adversarial Alignment Belongs in Sim-to-Real Sequence Policies

Robots trained in simulation fail on real hardware for two different reasons. The real world looks different: textures, lighting and sensor noise do not match the renderer. It also behaves differently: friction, mass and actuator lag do not match the physics engine. A common fix for the first problem is adversarial domain adaptation. A second network, the discriminator, tries to tell simulated features from real ones, and the feature extractor is trained to fool it. A common fix for the second problem gives the policy memory, a recurrent network or a convolutional network over recent history, so it can work out the current dynamics from how its own actions played out. This proposal argues that these two fixes conflict when they are stacked carelessly. If the discriminator sees the sequence model's summary of history, it pushes the policy to erase the very dynamics information that memory was added to recover.

The core idea fits in two sentences. Apply adversarial alignment only to per-time-step observation features, where a discriminator that sees one frame at a time cannot detect differences in dynamics. Then let a separate causal temporal module, left unaligned, integrate those features over time to infer the real system's dynamics. We test whether this placement outperforms both no alignment and whole-sequence alignment. We also test whether a dilated temporal convolutional network is a better choice than an LSTM for the unaligned history module when the evidence about dynamics arrives slowly.

Core Claim

The claim is structural. A frame-level discriminator is blind to dynamics by construction, because dynamics are only visible in how features change across time steps. A sequence-level discriminator, or one placed after a recurrent state, is not blind to them. A sequence-level discriminator should therefore cause the failure that A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies calls "blind alignment," where an encoder forced into full invariance loses the physical discernibility it needs to adapt its actions to real dynamics. That paper motivates this proposal: it supplies evidence that indiscriminate alignment harms control. It does not prescribe where alignment should sit in a temporal policy, and it does not test the frame-versus-history split, which is what we propose.

Write $o_t$ for the observation at step $t$, $a_t$ for the action, $z_t = f(o_t)$ for the per-frame encoder output, $m_t = g(z_{\le t}, a_{<t})$ for the causal history embedding, and $\pi(a_t \mid z_t, m_t)$ for the policy. Write $D$ for the discriminator, $d_t$ for the domain label of step $t$, and $\lambda$ for the alignment weight. The training objective is

$$\min_{f,g,\pi}\; \mathcal{L}_{\text{task}}(f,g,\pi) \;-\; \lambda\, \mathcal{L}_{D}\big(D(z_t), d_t\big), \qquad \max_D\; \mathcal{L}_D \text{ is replaced by } \min_D \mathcal{L}_D .$$

Here $\mathcal{L}_{\text{task}}$ is the reinforcement-learning or imitation loss, and $\mathcal{L}_D$ is the discriminator's cross-entropy loss. The adversarial gradient reaches $f$ through a gradient reversal layer, as used in FAN-Net: the layer passes features through unchanged on the forward pass and negates the discriminator's gradient on the way back. The history module $g$ receives no adversarial gradient.

The obvious objection is that per-frame features are not completely free of dynamics information. If real friction differs, the robot visits different states, so even single frames have different distributions in the two domains. We treat this as the main risk. The proposed mitigation is to condition the discriminator on proprioceptive state, so that it compares the appearance of frames that show the same physical configuration. This is tested as an explicit ablation.

Research Program

Recurrent policies trained with dynamics randomization appear in Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. That work trains an LSTM policy across randomized physics so that it infers dynamics online from state-action history. We reuse that recipe as the source of the history module's skill. Our contribution is to add appearance alignment on top without damaging that skill.

For appearance alignment we follow Adversarial discriminative sim-to-real transfer of visuo-motor policies, which aligns visuo-motor features adversarially. That paper also constrains our design: it reports that adversarial training needs a proportional-integral controller to tune the loss weight dynamically. We adopt that controller for $\lambda$ in every adversarial variant, so that differences in stability do not get mistaken for differences in placement.

For the history module we compare a gated LSTM with a temporal convolutional network. A temporal convolutional network is a stack of causal one-dimensional convolutions whose dilation doubles at each layer, so its receptive field grows exponentially with depth. Convolutional Sequence Modeling Revisited reports that LSTMs and GRUs stop retaining information beyond delays of roughly 50 to 200 steps. It also reports that convolutional stacks keep a short gradient path between distant steps, an argument developed for sequence-to-sequence models in Convolutional Sequence to Sequence Learning. This matters here for a specific reason. Some dynamics, such as slow thermal drift, payload changes or actuator lag under load, only reveal themselves over long windows. We therefore expect the advantage of the convolutional module to grow with how long that evidence takes to appear. The prediction is falsifiable, and we will report it as a curve rather than a single number.

The diagnostic instrument is a linear probe. In a controlled "pseudo-real" domain, the physics parameters are known even though the policy never sees them. We train linear readouts that predict friction, mass and latency from $m_t$, and separately from $z_t$. Blind alignment should show up as probe accuracy on $m_t$ falling under sequence-level alignment and staying intact under frame-level alignment. This turns the mechanistic claim from A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies into a quantity we can measure. We also measure the discriminator's held-out accuracy at each placement, to confirm that the appearance gap is actually closed.

Baselines

The naive reference point is a policy trained only in simulation, with neither randomization nor alignment, and deployed directly. It establishes the size of the gap. The first strong competitor is dynamics randomization combined with an LSTM policy and broad visual randomization, following Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. It is the strongest competitor because it addresses both gaps without any adversarial machinery or unlabeled real data. If it matches our method, the proposal has no reason to exist.

The second strong competitor is sequence-level adversarial alignment, in which the discriminator reads $m_t$, or reads windows of $z_t$, with the same proportional-integral weighting taken from Adversarial discriminative sim-to-real transfer of visuo-motor policies. This is the ablation that isolates our key ingredient, which is the placement of the discriminator, with everything else held fixed. A further ablation removes the state-conditioning from the frame-level discriminator.

The third competitor is co-training on simulated data mixed with the small labeled real set, as analyzed in A Mechanistic Analysis of Sim-and-Real Co-Training in Generative Robot Policies. The fourth is non-adversarial appearance normalization: we replace the discriminator with low-frequency Fourier amplitude normalization in the style of FAN-Net, which treats image style as low-frequency amplitude and keeps phase, and which reports degradation when more than a narrow band ($\alpha > 0.1$) is altered.

All methods share the same architecture backbone, simulator budget, unlabeled real-data pool, labeled real-data budget where one is used, and number of tuning trials. The search over $\lambda$ setpoints and randomization ranges is matched, and each method is evaluated over at least five seeds.

The file contains no task-success numbers for these methods on a shared benchmark. No reported result was found on Lacuna for real-robot success rates of dynamics randomization or of adversarial discriminative visuo-motor transfer that could serve as a fixed target. Baselines will therefore be re-run rather than quoted. Our method beats a baseline if it achieves higher success in the target domain with non-overlapping seed intervals, while its dynamics-probe accuracy on $m_t$ stays within noise of the unaligned randomization baseline.

Why not just…?

Why not just randomize appearance and dynamics and skip adversarial training? This is the most serious threat. Randomization works when the real world falls inside the randomized range. Adversarial alignment exploits unlabeled real frames, so it should help when the appearance gap lies outside that range, for example with a real camera's noise or a real object's texture. We test this directly by sweeping how far the target domain's appearance lies from the randomization range. If randomization alone stays equal to our method across the sweep, we will report that the adversarial component is unnecessary.

Why not just co-train on mixed sim and real data? Co-training needs labeled real trajectories, whereas frame alignment needs only unlabeled real frames. In addition, the mechanistic co-training paper itself finds that the type of alignment matters for preserving physical discernibility. We compare the two at matched real-data budgets. We expect co-training to win when labeled real data is plentiful, and our method to win when real data is scarce or unlabeled.

Why not just normalize the Fourier amplitude of images? It is cheap and non-adversarial. However, FAN-Net itself reports that it only tolerates a narrow low-frequency band, and it cannot handle gaps in sensor noise or geometry. It is a fair appearance-side baseline, and if it ties with our method we will recommend it instead.

Why not just use an LSTM, as prior sim-to-real work does? For short-horizon dynamics an LSTM may suffice, and we expect the two to tie there. The memory limits reported in Convolutional Sequence Modeling Revisited predict that the convolutional module wins only when the evidence about dynamics spans more than about a hundred steps. The horizon sweep will show whether that holds.

Proposed Experiments

The primary testbed is a simulated pushing task and a reaching task on a robot arm, matching the task style of the dynamics-randomization and visuo-motor transfer papers. It has two target domains. The first is a sim-to-sim "pseudo-real" domain: a different renderer combined with physics parameters held out from the randomized range, where ground-truth dynamics are available for probing. The second is a small real-robot evaluation. To test long-horizon dynamics, we inject latent dynamics that only become observable after a controllable delay of 20 to 500 steps, such as slowly changing friction or delayed actuation.

The metrics are success rate and return in the target domain, probe accuracy for the dynamics parameters on $m_t$ and $z_t$, discriminator accuracy, and training stability measured by the variance of $\lambda$ and of the losses across seeds.

The decisive check has three parts. Sequence-level alignment must lower probe accuracy on $m_t$ and lower target-domain success compared with frame-level alignment. Frame-level alignment must beat dynamics randomization alone once the target appearance lies outside the randomized range. The convolutional history module's advantage over the LSTM must grow with the delay before the dynamics become observable. If the first part fails, the blind-alignment explanation does not apply to temporal policies. If the second fails, the adversarial component is unneeded. Either outcome would be reported as the main finding.