Visible Excitation: Choosing What a Visual Simulator-Calibration Discriminator Sees

Robots trained in simulation often fail on real hardware. One common fix is to calibrate the simulator's physics: record the real robot, then adjust mass, friction, and damping until simulated motion matches the recording. A recent method, EASI, frames this calibration as a contest. A small neural network, the discriminator, tries to tell real state transitions from simulated ones, and an evolutionary search adjusts physics parameters until the discriminator cannot. This works when the robot's state arrives as a clean vector. Many interesting scenes only arrive as camera images, though: piles of dice, poured material, cluttered trays. Once the discriminator looks at pixels, two old problems return together. It can separate sim from real by rendering differences such as lighting and texture instead of motion. And the recorded motion may not reveal the physics at all, either because the robot barely moved or because the movement fell outside the patch of image the network actually attends to.

This note proposes to treat those two problems as one measurable quantity. Control theory already has a name for "the data moved enough to reveal the system": persistent excitation, a rank condition saying the recorded signals vary in enough independent directions to pin down every unknown. We define a visual version of it. The condition asks whether, inside the image window the discriminator can see (its receptive field), changes in physics parameters produce feature changes that are large and independent, relative to the changes produced by rendering nuisances. This score can be computed entirely in simulation, before touching hardware. It then serves two purposes: it picks the discriminator's receptive field, and it picks the probing motion the robot executes to collect its one real calibration trajectory.

Core Claim

In image-based adversarial simulator identification, parameter recovery succeeds or fails according to a single simulation-computable quantity: the smallest physics-to-nuisance sensitivity ratio of the discriminator's features along the probing trajectory. Choosing the receptive field and the probe to maximize this quantity should beat both a global-view discriminator and a hand-tuned local one, and it should match an oracle sweep, without real-robot trial and error.

The quantity is defined as follows. Let $\theta$ be the physics parameters and $\eta$ the rendering parameters (lighting, texture, camera jitter). Let $\phi_r(t)$ be the discriminator-input features at time $t$, meaning local space-time patches of the frame-difference stack, pooled over a receptive field of size $r$. Let $J_\theta(t)=\partial \phi_r(t)/\partial\theta$ and $J_\eta(t)=\partial \phi_r(t)/\partial\eta$, both estimated by finite differences in the simulator along a candidate probe $\tau$. The visual excitation score is the smallest generalized eigenvalue

$$ \kappa(r,\tau)=\lambda_{\min}\Big(\textstyle\sum_t J_\theta(t)^\top J_\theta(t),\;\sum_t J_\eta(t)^\top J_\eta(t)+\epsilon I\Big), $$

where $\epsilon$ is a small regularizer. Without the nuisance term, this is the sensitivity analogue of the full-rank Hankel condition in Willems' Fundamental Lemma as made quantitative by Berberich et al.. A near-zero $\kappa$ means some physics direction is either invisible at scale $r$ or indistinguishable from a rendering change. In both cases the discriminator gives the search no useful signal along that direction.

The receptive field enters in two opposing ways. If $r$ is too small, the window cannot contain the object's displacement over the frame horizon. This is the classical aperture problem of motion estimation, and it is a literal instance of the local receptive field limitation. If $r$ is too large, the window takes in scene layout and background, where renderer differences dominate. Jaipuria et al. make exactly that point for image translation.

Relation to Prior Work

Four lines of work shape the design, each in a different role.

On the Role of Receptive Field in Unsupervised Sim-to-Real Image Translation motivates the proposal. It sweeps a translation discriminator's receptive field from 46×46 down to 7×7 and shows a trade-off: large fields warp synthetic geometry, and very small fields give a weak realism signal. The choice there is made by trial and error on static images. We keep the finding and replace the trial and error with a criterion computed in simulation, and we apply it to dynamics rather than appearance.

EASI supplies the method we reuse, including evolutionary search and a Wasserstein discriminator used as the fitness signal. Its own limitation, as the file notes, is that it depends on the real trajectory being informative and processes only state vectors. TuneNet estimates parameter corrections from paired sim and real video. It likewise produces zero updates when the motion is uninformative, and it never examines its encoder's spatial scale.

Learning Active Task-Oriented Exploration Policies supplies the idea of weighting excitation by task relevance, which cut pouring error from 22g to 14g. Its setting assumes scalar, fully observed parameters, so it never faces the question of whether a probe is visible to the perception system.

The control literature constrains the design. The "paradox of control success" in data-driven MPC and the probing-cost trade-off in data-enabled policy optimization both warn that probing must be short and targeted. That is why we restrict ourselves to a single short probe chosen offline rather than continuous dither.

Research Program

The work has three stages.

Stage 1: the frame-difference discriminator. We build a discriminator that sees only local space-time patches of frame differences. Each patch is normalized so that static appearance largely cancels, and each is conditioned on the commanded action. A pyramid of patch extractors provides receptive fields from roughly 8 to 128 pixels, with the global view as a special case.

Stage 2: computing the score and choosing the receptive field. We compute $\kappa(r,\tau)$ in simulation by perturbing each physics parameter and each rendering parameter around a nominal value. The receptive field is chosen as $r^\star=\arg\max_r \kappa(r,\tau)$.

Stage 3: choosing the probe. Over a library of parameterized probe motions, such as pushes, lifts, and shakes with varying amplitude and location, we select the probe that maximizes $\kappa$. The objective is weighted by a task-relevance matrix, in the spirit of Liang et al.: the sensitivity of downstream task cost to each parameter, estimated in simulation. It is subject to a limit on actuator effort. The robot then executes that one probe, and EASI-style search runs with the chosen discriminator.

The analysis question is whether $\kappa$ predicts the outcome. Across a grid of receptive fields and probes, the method should be validated by rank correlation between $\kappa$ and the final parameter error, not only by the best achieved number.

Baselines

Uniform domain randomization. The naive reference point trains the downstream policy under uniform randomization of physics and rendering, with no calibration. EASI reports that its identification reduces real Cartpole pole-angle error by about 37% and cart-velocity error by about 25% relative to this baseline (EASI). We rerun it with the same policy-training budget as every other arm.

State-based EASI. EASI run on ground-truth state vectors is the strongest existing competitor. On TriFinger these come from the platform's 6D object tracking. It sees perfect state and no rendering gap, so it is effectively an upper bound wherever state is available. Beating it is not the goal on single-object tasks. Matching it within noise from pixels alone would count as success there, and exceeding it on multi-object scenes where no tracker exists would be the real win. No reported result was found on Lacuna for EASI with image observations.

Global-view and Jaipuria-style discriminators. A global-view EASI discriminator (full frame, same parameter count) tests the shortcut failure. A Jaipuria-style discriminator with one fixed mid-sized receptive field (16×16 equivalent) tests whether a sensible default already suffices.

Oracle sweep. An oracle sweep picks the receptive field in hindsight using real-robot parameter error. Our method is not expected to beat it. It should come within the sweep's seed-to-seed spread while using one real trajectory instead of one per candidate.

TuneNet on video. TuneNet run on the same video represents the non-adversarial alternative. It reports 87% bounce-shot success in its own setting; that number is not directly comparable to ours but anchors expectations.

Ablations. Three ablations isolate the ingredients: the chosen receptive field with a random probe from the library, the chosen probe with a fixed receptive field, and $\kappa$ without the nuisance term, which is plain sensitivity.

Fairness controls. All arms use the same real trajectory budget (one probe, except the oracle), the same evolutionary search budget, the same discriminator capacity, and five seeds.

Why not just…?

Why not just track object poses and run EASI on states? Where a reliable tracker exists, this is a real threat, and we treat it as the ceiling rather than argue against it. The proposal's value is confined to scenes that lack a low-dimensional state. The RRC 2021 25-dice task, which no team solved, is our target example, along with poured or granular material. If image-based calibration cannot approach state-based EASI on single-cube TriFinger, the method fails its first sanity check.

Why not just translate simulated images to look real first, then calibrate? Translation is exactly where Jaipuria et al. found geometry warping. A translator that alters object outlines would corrupt the motion the discriminator must judge. We include a translate-then-global-discriminator arm to test whether this cost appears in dynamics as well as in static shapes.

Why not just feed frame differences to a global discriminator? Differencing alone may remove most static rendering gaps. If so, the receptive field choice matters less than we claim. The global-view-on-differences arm is the direct test. We expect it to still pick up camera jitter and background flicker, which the nuisance term in $\kappa$ explicitly accounts for, but this is an empirical question the experiment settles.

Why not just inject PRBS dither? A pseudorandom binary sequence (PRBS) is a standard broadband test signal for exciting a system. It excites actuators with no regard to what the camera can see. Liang et al. already show that task-agnostic probing can do no better than random on box dragging. The random-probe ablation measures how much of the gain comes from choosing the probe for visibility.

Proposed Experiments

Sim-to-sim tests with known ground truth. We use rendered-camera versions of Isaac Gym Cartpole and Ballbalance. The "real" domain is a second renderer with different textures, lighting, and camera noise, plus held-out physics parameters. Physics error can then be measured exactly, and rendering shift varied on purpose.

Real-hardware tests. We then move to a camera-observed real Cartpole, mirroring EASI's hardware check, and to TriFinger through the Real Robot Challenge cloud. TriFinger has 10 Hz cameras and ground-truth object tracking for evaluation only. It covers single-cube pushing and a reduced multi-dice pushing task.

Metrics. We measure parameter error (in sim-to-sim), multi-step transition error in tracked pose space, and downstream task success of policies trained in each calibrated simulator.

Decisive checks. Three results decide the proposal. First, the rank correlation between $\kappa$ and parameter error across the receptive-field-by-probe grid must be clearly positive. We would treat a Spearman correlation below 0.5 as falsifying the central claim. Second, in sim-to-sim, increasing rendering shift must leave our estimates stable while degrading the global-view arm. Third, on multi-dice TriFinger, where state-based EASI is unavailable, our calibrated simulator must yield higher real success than domain randomization. If $\kappa$ fails to predict outcomes, the proposal reduces to a receptive-field sweep, and we would report it as such.