Does Action Geometry Change What Synthetic Robot Demonstrations Are Worth?

Robot manipulation policies learn from a mixture of cheap simulated demonstrations and expensive real ones, but nobody knows the exchange rate between the two. In video tracking, On the Equivalency, Substitutability, and Flexibility of Synthetic Data measured roughly 30 synthetic frames per real frame, and found that 60–80% of real data could be replaced by synthetic pretraining. For manipulation, no comparable number exists. We propose to measure it and test one specific lever that might move it: how the policy represents the gripper's orientation. Most diffusion policies treat a 3D pose as a flat vector of 12 numbers. A diffusion policy is a generator that produces actions by gradually denoising random noise. When it adds and removes noise from those 12 numbers, it produces "rotations" that are not valid rotations. The Lie We Tell showed that doing the diffusion directly on the space of rigid motions keeps every intermediate pose valid, cuts orientation jitter by about 10×, and improves results in simulation and on hardware.

Our claim has two parts. First, part of the sim-to-real gap in manipulation comes from geometry: synthetic demonstrations often rotate the gripper along paths that real human operators never take. Second, a policy whose action space respects rotation geometry should both need fewer real demonstrations after synthetic pretraining and be less hurt by these artifacts. The study is a controlled measurement of substitutability curves. We vary the action representation, the amount of real data, and the geometric quality of the synthetic data independently, so that the model-side and data-side explanations can be told apart.

Core Claim and What Would Be New

The prior work leaves a gap. The Lie We Tell supplies the method, its Lie Diffuser Actor. That model injects noise as small velocity "twists" in the tangent space of the rigid-motion group and maps them back through the exponential map, so every pose stays valid. The paper treats its training data as fixed and never asks how much each data source is worth. On the Equivalency, Substitutability, and Flexibility of Synthetic Data supplies the evaluation protocol but works only with 2D bounding boxes, where there is no geometric constraint to violate. That paper also motivates our geometric hypothesis. It found that the structure of synthetic data mattered more than its volume: simple synthetic walking hurt DanceTrack, while behaviorally complex multi-group scenes raised HOTA from 68.8 to 73.5. We ask whether rotational structure plays the same role in manipulation.

LIBERO constrains the design. It found that naive pretraining on 90 simulated tasks hindered later learning, so we cannot assume synthetic pretraining helps at all. The substitutability curve may be flat or negative for some representations, and the protocol must be able to show that.

We define the equivalency ratio at a target success rate $s$ as

$$\rho(s) = \frac{N_{\text{syn}}(s)}{N_{\text{real}}(s)},$$

where $N_{\text{real}}(s)$ is the number of real demonstrations a policy trained from scratch needs to reach success $s$, and $N_{\text{syn}}(s)$ is the number of synthetic demonstrations that yields the same $s$ when combined with a small fixed real set. Both quantities are read off interpolated learning curves. Substitutability is the largest fraction of the full real set that can be removed, given synthetic pretraining, while staying within a bootstrap confidence interval of the all-real policy. The new result would be the first measured $\rho$ and substitutability envelope for manipulation, and the first evidence on whether action geometry shifts them. No reported result was found on Lacuna for an equivalency ratio in robot manipulation.

Research Program

We build a digital twin in simulation of four manipulation tasks matching the hardware suite in The Lie We Tell: Move Doll Platform, Put Block in Box, Sort Blocks, and Stack Cups. In the twin, a scripted motion planner generates synthetic demonstrations. The planner has one controlled knob, the rotational interpolation between waypoints. In the "geodesic" setting it rotates along the shortest rotation path, following screw motions. In the "artifact" setting it interpolates each Euler angle linearly, a common shortcut that produces valid but unnatural and sometimes jerky rotations. A second knob adds small orientation jitter on the manifold. The real data are teleoperated demonstrations on a physical arm.

We also compute a geometric gap statistic between the synthetic and real demonstration sets: the average geodesic deviation of each demonstration's orientation path from the shortest rotation path between its start and end, compared across the two sources. This tells us whether the artifact knob moves synthetic data toward or away from real behavior.

Four policy variants share the same perception encoder, transformer backbone, diffusion schedule, and parameter count, and differ only in the action head. The first is Euclidean diffusion over the 12-number pose, as in 3D Diffuser Actor. The second is the same model with an SVD projection onto valid rotations at the final step only. The third regresses a continuous, low-dimensional rotation encoding with no manifold constraint during denoising. The fourth is the Lie Diffuser Actor. The comparison between the second and fourth variants isolates the key ingredient: whether it matters that intermediate denoising steps stay on the manifold, or only that the output is valid.

For each variant and each synthetic condition, we pretrain on synthetic budgets spanning roughly an order of magnitude and fine-tune on real budgets of 5, 10, 20, 40 and 80 demonstrations per task. We run three seeds and use 20 hardware rollouts per cell for the headline conditions. This compute is expensive, so the full grid runs in a sim-to-sim proxy first. There, "real" means human teleoperated demonstrations in simulation (LIBERO's teleoperated demonstrations, and CALVIN environment D as the target), and "synthetic" means planner-generated demonstrations. Hardware then confirms the curves at the three most informative real budgets.

Baselines

The naive reference point is a policy trained only on real demonstrations, with no synthetic data, at each real budget. Every substitutability claim is measured against this curve. If synthetic pretraining does not beat it at matched real budgets, the synthetic data has no value for that representation, which is what LIBERO's result warns may happen.

The strongest existing method is the Lie Diffuser Actor from The Lie We Tell, trained on the fixed data mixtures used there. It reports an average chain length of 3.51 on CALVIN ABC→D against 3.27 for the Euclidean 3D Diffuser Actor. It also reports 94.13% against 93.87% on LIBERO-Long with OpenVLA, and 75% against 55% on hardware Sort Blocks. We will first reproduce the CALVIN gap under our shared encoder to confirm the implementation. Beating it in our sense does not mean a higher peak success rate. It means showing that the Lie head's advantage grows as the real budget shrinks, which would mean a lower $\rho$ and higher substitutability. A constant offset at all budgets would mean geometry helps but does not change the exchange rate.

The final-step SVD projection variant is the ablation that isolates on-manifold denoising. All variants use the same data, the same number of gradient steps, and the same hyperparameter search budget of eight configurations tuned on a held-out simulated validation split, with the tuning done separately for each variant. Hardware evaluators are blind to the variant.

A data-side baseline completes the set. The Euclidean policy is trained on synthetic data whose rotations were re-interpolated geodesically after generation. This directly tests whether cleaning the data substitutes for changing the model.

Why not just…?

Why not just project the output onto valid rotations with SVD? The file's evidence against this is indirect. Projection after unconstrained steps distorts conserved quantities in dynamical integration, as shown in Energy-preserving methods on Riemannian manifolds. In diffusion, The Lie We Tell reports that intermediate steps stay off the manifold and that jitter is about 10× higher. But that paper does not compare against projection at matched data scarcity, so the objection remains open. Our projection-at-output variant settles it. If its curves coincide with the Lie head's, the intermediate-step geometry does not matter for data efficiency, and we will report that.

Why not just clean the synthetic data instead of changing the model? This is the most serious threat. If geometric artifacts live in the planner, fixing the planner may be enough. The geodesically re-interpolated data baseline tests it directly. A model-side effect exists only if the Lie head outperforms the Euclidean head on clean synthetic data, and degrades less as the artifact knob increases. If cleaning closes the whole gap, the paper's contribution becomes a data-quality result: rotational structure is the manipulation analogue of the behavioral complexity that On the Equivalency, Substitutability, and Flexibility of Synthetic Data found in tracking. That would still be a useful finding.

Why not just add more synthetic data and randomize the simulator more? Volume alone has failed before. LIBERO pretraining on 90 tasks hurt later learning (LIBERO), and simple synthetic data hurt DanceTrack in the tracking study. Our synthetic-budget axis measures directly whether each curve saturates. If the Euclidean curve keeps rising and eventually matches the Lie curve, geometry changes the rate of improvement but not the ceiling. The equivalency ratio $\rho$ captures exactly that distinction.

Why not just use a better continuous rotation encoding? A smooth low-dimensional rotation encoding removes discontinuities without Lie group machinery. We include it as a variant. The file contains no reported manipulation result for it, so it could plausibly match the Lie head. If it does, the cheaper option wins.

Proposed Experiments and Decisive Checks

The primary check is an interaction effect between representation and real budget. The difference between the Lie and Euclidean policies should be largest at 5–10 real demonstrations and shrink by 80. The test fits a logistic model of success on representation, log real budget, and their interaction, with bootstrap intervals over rollouts. A zero interaction falsifies the exchange-rate claim, even if Lie remains better overall.

The second check is a dose-response on the artifact knob. As Euler-interpolation artifacts increase, the Euclidean policy's substitutability should fall faster than the Lie policy's. Across conditions, the size of that difference should track the geometric gap statistic.

The third check stratifies tasks by total rotation angle per demonstration. The Lie We Tell notes that on translation-heavy Put Block in Box the Euclidean head held a slight edge. We therefore predict no effect on that task and the largest effect on Sort Blocks and Stack Cups. An effect that appeared uniformly across tasks would point to a confound, such as tuning differences, rather than geometry.

A secondary check revisits LIBERO's pretraining result. We repeat LIBERO-90 pretraining followed by LIBERO-Long adaptation with both heads, to see whether the harm from pretraining is smaller when rotations stay valid. This is a cheap simulation-only test of whether geometric bias contributes to that failure.