Counting Information, Not Frames, in Cross-Embodiment Robot Data

Robot learning now routinely pools demonstrations from many sources: human videos, several robot arms, humanoids. Practitioners measure and mix these sources by how many frames or steps each contains. Consecutive frames of a smooth motion, however, are nearly copies of one another. A dataset recorded at a high control rate, or full of slow approach phases, can hold ten times the frames of another dataset without holding ten times the evidence. Learning theory has a name for this. The mixing time of a data stream is roughly how many steps it takes for the stream to forget where it was, and a stream of $N$ correlated samples carries about $N/\tau$ independent samples' worth of information, where $\tau$ is that forgetting time. We propose to make this "effective sample size" the unit by which cross-embodiment data is measured, weighted, and compared.

The concrete claim is this. When heterogeneous embodiments are pooled, weighting each source by its nominal frame count systematically over-represents slow-mixing, highly redundant sources. A cheap estimate of each source's effective sample size can fix this. That estimate is the integrated autocorrelation time of features the policy itself computes, a standard quantity from Markov chain Monte Carlo practice. Correcting for redundancy also makes a cleaner scientific question answerable: once temporal redundancy is factored out, how many effective human-video samples is one effective robot sample worth? That residual "exchange rate" is a direct, measurable size of the embodiment gap. Whether the correction matters at all is itself open. Theory for the square loss says the correlation penalty may vanish at leading order, while results for non-convex models say it persists. Our experiments are designed to tell these apart for real behavior-cloning policies.

What the two literatures give us, and what they leave open

The dependent-learning literature motivates the proposal. Least Squares Regression with Markovian Data proves the minimax error for linear regression on a Markov stream is $\Theta(\tau_{\text{mix}}\sigma^2 d/T)$, exactly $\tau_{\text{mix}}$ times worse than independent data. It also shows that uniform experience replay improves bias decay only from $1/\tau_{\text{mix}}$ to $1/\sqrt{\tau_{\text{mix}}}$. Shuffling a buffer therefore decorrelates minibatches without creating information. The spectral-gap-discounted bounds in Adaptive Huber Regression on Markov-dependent Data and the deep-network PAC-Bayes rates in Minimax optimality of deep neural networks on dependent data via PAC-Bayes bounds point the same way.

One result constrains the proposal and could undercut it. Sharp Rates in Dependent Learning Theory shows that under square loss, with realizability and moment conditions, the mixing penalty moves into an additive burn-in term, and the leading rate matches independent data. If modern policies sit in that regime, effective-sample weighting should help only at small data scales. We treat this as a falsifiable prediction, not something to argue around.

On estimation, Empirical PAC-Bayes bounds for Markov chains estimates a pseudo-spectral gap from one trajectory but needs discrete transition structure. Unknown mixing times in apprenticeship and reinforcement learning avoids knowing the mixing time through coupling from the past, at a cost of $O(|S|T_{\text{mix}})$, and assumes expert and learner share one decision process. Neither scales to continuous, multi-embodiment manipulation data. That is why we use a cruder but scalable autocorrelation estimator.

The embodiment literature supplies the setting. Open X-Embodiment pools 60 datasets from 22 robots and reports positive transfer with RT-1-X and RT-2-X. Nothing in this file says its mixture weights account for temporal redundancy. Any-point Trajectory Modeling (ATM) pretrains a point-track predictor on action-free human and robot video. Flow as the Cross-Domain Manipulation Interface uses predicted object flow as an embodiment-invariant interface. Both train on dense, smooth tracks counted by frame. Neither measures how much independent evidence those tracks contain.

The closest work at the intersection is Mixing-Time Regularized Policy Gradient. It penalizes hitting time to escape slow-mixing plateaus in policy gradient, but only on a two-state MDP, and it changes the policy, not the data mixture. We do the reverse: we leave the learner alone and change how data from different bodies is counted.

Core Claim

For a source $e$ (one embodiment or dataset) with $N_e$ frames, define its effective size $N^{\text{eff}}_e = N_e/\hat\tau_e$. Here $\hat\tau_e$ is the integrated autocorrelation time of a scalar or low-dimensional per-frame feature sequence along trajectories:

$$ \hat\tau_e = 1 + 2\sum_{k=1}^{K}\hat\rho_e(k), $$

where $\hat\rho_e(k)$ is the average lag-$k$ autocorrelation of that feature within trajectories of source $e$, and $K$ is a truncation lag chosen by the standard self-consistent window rule. We claim three things. First, sampling sources in proportion to $N^{\text{eff}}_e$, and subsampling within a source at stride about $\hat\tau_e$ when the data budget is limited, beats frame-proportional mixing whenever sources differ substantially in $\hat\tau_e$. Second, it matches or beats episode-proportional mixing. Third, policy success, plotted against cumulative effective samples, collapses onto a single curve across sources better than against frames or episodes. The constant offset that remains between human and robot sources on that curve is our exchange rate.

Research Program

The work has three stages.

The first stage validates the estimator where the truth is known. On the Gaussian autoregressive least-squares setup of Least Squares Regression with Markovian Data, $\tau_{\text{mix}}$ is set by the spectral radius. We check that $\hat\tau$ computed on features recovers it, and that effective-sample weighting reproduces the predicted error ordering when two AR sources with different $\tau$ are mixed.

The second stage chooses the feature. Candidates are three: frozen visual-encoder embeddings, the policy's per-frame loss, and a random projection of the per-frame gradient on the last layer. We pick by stability across seeds and by agreement with the controlled ground truth in stage three. Because the estimator only needs a forward or backward pass over trajectories already in the dataset, its cost is one epoch. This is the scalability that the PAC-Bayes and coupling estimators lack.

The third stage applies the weighting to the embodiment settings below and measures the exchange rate.

Baselines

The naive reference point is frame-proportional mixing with uniform shuffled replay. This is what pooling all trajectories into one buffer does by default, and the dependent-learning results above predict it is miscalibrated.

The strongest simple competitor is episode-proportional mixing, where each trajectory counts once. It is a crude correction for redundancy that every practitioner already has. Effective-sample weighting must beat it, not just frames, for the proposal to matter.

A second strong competitor is control-rate normalization: resample every source to a common frequency, then mix by frames. This removes the most obvious source of redundancy differences between embodiments.

For the downstream policies, the strongest existing methods in the file are ATM on LIBERO-Long and RT-1-X on the Open X-Embodiment mixture. Any-point Trajectory Modeling reports 39% success on LIBERO-Long, against 16% for plain behavior cloning. Open X-Embodiment reports RT-1-X achieving a 50% higher success rate than baselines on small-data domains. The file gives no per-dataset mixture weights or absolute per-task numbers for RT-1-X, so for an exact mixture baseline there is no reported result on Lacuna, and we re-run it ourselves.

Every arm uses the same architecture, the same total gradient steps, the same raw data, and the same hyperparameter search budget. Only the sampling distribution over sources and the within-source stride change. Beating a baseline means a gain in success rate larger than the across-seed spread, using at least five seeds, at matched compute.

The key ablation keeps effective-sample weights but swaps $\hat\tau_e$ for random values with the same spread. This isolates whether the gain comes from the specific redundancy estimate rather than from merely flattening the mixture.

Why not just…?

Why not just weight by episode count? This is the real threat. If sources differ only in recording rate and episode length, episodes and effective samples nearly coincide, and our method adds little. They come apart when episodes contain different amounts of internal variety, such as long static grasps versus continuous pouring, or a single long human video versus many short robot clips. The controlled duplication and long-static-phase experiments below are built to create exactly these cases. If episode weighting matches us on the real Open X-Embodiment subsets, we will report that the cheap proxy suffices.

Why not just resample everything to a common control rate, or predict action chunks? Resampling removes redundancy caused by frequency, but not redundancy caused by the task itself, since a slow approach motion is redundant at any rate. Chunking changes what the policy outputs, not how much independent evidence the data holds. The file's evidence on chunking is indirect: ATM already predicts multi-step tracks and still reaches only 39% on LIBERO-Long. Our long-static-phase experiment tests whether rate normalization leaves a gap that $\hat\tau$ closes.

Why not just shuffle a big replay buffer? Shuffling breaks correlation within a minibatch but does not change how often each source is sampled, and it creates no information. Least Squares Regression with Markovian Data shows replay still scales with mixing time. LIBERO reports that replay can even hurt forward transfer relative to sequential fine-tuning. Shuffled replay is our naive baseline, so the comparison is direct.

Why not just rely on scale, given the burn-in result? If Sharp Rates in Dependent Learning Theory describes these policies, our gains should shrink as total data grows. We measure gain against dataset size explicitly. A gain that vanishes with scale would be a useful empirical confirmation of the burn-in regime for non-convex policies, which the theory leaves open.

Proposed Experiments

The first decisive check is a controlled duplication test on LIBERO. Take two LIBERO-Object task groups. Make one artificially slow-mixing by temporally interpolating its demonstrations to several times their frames, or by padding with near-static frames; this adds frames without adding information. Under frame weighting, the inflated group should crowd out the other, and the other's success should drop. Under effective-sample weighting, $\hat\tau$ should rise roughly in proportion to the inflation, and both groups' success should stay near the uninflated reference. Episode weighting should pass the interpolation test but fail the static-padding variant only if padding is applied unevenly across episodes. We design one variant of each kind.

The second check uses human-plus-robot pretraining for ATM on LIBERO-Long. We pretrain the track predictor on mixtures of human video and LIBERO robot video under frame, episode, rate-normalized, and effective-sample weighting. Downstream success is compared against the reported 39%. We then vary the human-video fraction and fit success against human and robot effective samples. The ratio at which they trade off is the human-to-robot exchange rate. We repeat the fit using frames instead of effective samples, to show how much of the apparent embodiment discount was really redundancy.

The third check is a cross-robot mixture: a several-dataset subset of Open X-Embodiment chosen to span different control rates and task tempos. On it we train an RT-1-X-style policy under the four weighting schemes and report held-out success in simulation, plus a modest real-robot evaluation if hardware allows.

Across all settings, the scientific test is curve collapse. If success against effective samples aligns across sources clearly better than against frames or episodes, measured by residual variance of a shared saturating fit, then temporal redundancy is a real, separable part of what makes embodiments look unequal. If it does not align, the proposal's central premise fails, and we report that.