When Discrete Action Tokens Stop Meaning Anything: Identifiability Repair for Video-Derived Latent Actions

A popular recipe for teaching robots from video first compresses each pair of video frames into a discrete "latent action token", an entry from a small learned vocabulary that summarizes what changed between the frames. A separate decoder then translates tokens into motor commands. The vocabulary acts as the bridge between what a camera sees and what a robot arm does. This proposal argues that the bridge can fail quietly. Two different tokens can decode into robot behavior that no observer could tell apart, and the decoder can learn to ignore the token entirely, acting on the robot's own joint readings and history instead. Usage statistics stay healthy and offline losses stay low, yet the distinctions the video model draws are lost before they reach the motors.

We propose to measure this failure directly and then repair it. The measurement executes each token in simulation and uses a statistical test to decide which tokens produce indistinguishable behavior. Tokens that fail the test are treated as "tied", in the same sense that retrieval research treats items at equal Hamming distance as tied. The repair fine-tunes the token-to-action decoder with adversarial imitation, a method in which a learned critic rewards the robot for producing transitions that look like expert transitions carrying the same token. A second reward term pays the robot when its behavior lets a classifier recover which token it was given. The central claim is that the number of tokens with distinguishable behavior predicts long-horizon task success better than codebook usage does, and that raising that number with on-policy feedback improves success over the strongest video-to-action system in our research file.

Core Claim

Latent-variable modeling has a precise account of when latents get ignored. Posterior Collapse and Latent Variable Non-identifiability proves that posterior collapse (the encoder's distribution over latents falls back to the prior, so latents carry no information) happens if and only if distinct latent values produce the same data distribution under the decoder. Discrete tokenizers are often credited with removing collapse. Neural Discrete Representation Learning introduced the vector-quantized autoencoder, and Discretized Bottleneck in VAE shows that such codebooks avoid the continuous KL collapse in text models. Those papers address the encoder side. In a two-stage robot system, the identifiability that matters sits downstream. Token $c_1$ and token $c_2$ must lead to different action distributions once a robot decoder runs them in the world. Nothing in the vector-quantization objective enforces this, because the decoder is trained afterward by behavior cloning, and it can exploit proprioception just as an autoregressive text decoder exploits its own history.

We make three claims, each testable. The first is diagnostic: video-derived latent action codebooks of the kind used in Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining (DeFI) contain large groups of behaviorally tied tokens, even when every token is used. The second is predictive: across checkpoints and codebook sizes, the effective number of distinguishable tokens correlates with long-horizon success more strongly than codebook perplexity does. The third is interventional: on-policy adversarial fine-tuning with a per-token identifiability reward increases the effective token count and improves success. Off-policy fixes cannot see behavioral ties, because ties exist only in executed rollouts.

How the four directions fit

The cross-modal gap is the setting. Video embeddings and motor commands live on very different manifolds, and DeFI's discrete latent actions are its answer to that gap. Posterior collapse supplies the theory that defines failure as non-identifiability. Discrete distance ambiguity supplies the measurement. Hashing as Tie-Aware Learning to Rank shows that ranking metrics silently break ties at random and introduces tie-aware average precision ($AP_T$), the expected precision over all orderings of tied items. Hashing with Uncertainty Quantification via Sampling-based Hypothesis Testing declares items equivalent through a sampling-based test. We use the test to define behavioral ties and $AP_T$ to score how well a token can be recovered from a rollout without rewarding arbitrary tie-breaking. Adversarial imitation supplies the repair. Behavioral ties only appear when the robot acts, so the fix must learn from the robot's own rollouts. InfoGAIL already pairs an adversarial reward with a posterior network $Q(c\mid s,a)$ that recovers a latent code. Its codes, however, are a small user-specified prior, such as a two-way one-hot driving style. Its posterior is trained jointly with the policy, so it cannot reveal that a large pretrained vocabulary has collapsed.

Several papers constrain the design. Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning (SAQ) shows that discrete action codes stabilize offline policy learning, with roughly 44% success on Robomimic Square where continuous CQL failed. This is evidence that discreteness is worth keeping rather than designing away. UniCode fixes codebook under-use with exponential-moving-average synchronization. That is precisely the kind of usage fix we expect to leave behavioral ties untouched. Latent Wasserstein Adversarial Imitation Learning shows that adversarial imitation becomes sample-efficient when the critic works in a well-structured latent space. We borrow the idea of conditioning the critic on a pretrained representation. Unlike that paper, our representation is discrete and comes from video.

Research Program

Diagnostic. We reimplement a DeFI-style pipeline: an inverse dynamics vector-quantized encoder that maps frame pairs $(o_t, o_{t+n})$ to one of $K$ tokens, plus a decoder trained by behavior cloning that maps token and robot state to an action chunk. For each token, from a matched set of start states, we roll out the decoder in simulation and collect the resulting state transitions. For each pair of tokens, a two-sample test with a held-out classifier decides whether the two rollout distributions are distinguishable at level $\alpha$, with Bonferroni correction across pairs. We then form equivalence classes by taking connected components of the "indistinguishable" graph. The effective token count $K_{\text{eff}}$ is the number of these classes. Tie mass is the fraction of expert-labeled transitions whose token falls in a class larger than one. We also train a fixed posterior $Q(c \mid s, s')$ on decoder rollouts and score it with tie-aware $AP_T$. When $Q$ assigns tied tokens equal scores, $AP_T$ charges the expected rank rather than a lucky one. These quantities are computed alongside codebook perplexity across $K \in \{16, 64, 256\}$ and across training checkpoints.

Repair. We fine-tune the decoder $\pi_\theta(a \mid s, c)$ by reinforcement learning with the per-step reward

$$r(s, s', c) = -\log\big(1 - D_\psi(s, s', c)\big) + \lambda \log Q_\omega(c \mid s, s').$$

Here $D_\psi$ is a critic trained to separate expert transitions, labeled with the token the frozen video encoder assigns them, from policy transitions generated under token $c$. $Q_\omega$ is a token posterior trained on policy rollouts. $\lambda$ trades occupancy matching against identifiability. The first term keeps behavior under each token close to what the expert did when the video model saw that token. The second term penalizes ties, because tokens with indistinguishable behavior cap $\log Q_\omega$ at the log of the inverse class size. The video encoder stays frozen throughout, so any gain comes from the motor side honoring distinctions the video model already makes.

We test a tie-aware variant in which the second term is replaced by a smoothed $AP_T$ of the true token's rank under $Q_\omega$, using the soft histogram relaxation of Hashing as Tie-Aware Learning to Rank. We treat this as an ablation, not the headline. Plain mutual information may be enough, and the experiment should say so.

Analysis. We trace which tied tokens the repair separates and check whether they correspond to visually distinct transitions, using nearest frame pairs from the expert data. We also test whether the gains come from a proprioception shortcut being broken. The probe is the decoder's sensitivity to the token: the change in its output when the token is swapped while the state is held fixed. This is the motor-control analogue of the vanishing $\nabla_z \log p_\theta(x\mid z)$ that defines collapse.

Baselines

The naive reference point is the behavior-cloned DeFI-style decoder with no fine-tuning, run from our reimplementation with identical tokenizer, data, and backbone. Its reported counterpart in Disentangled Robot Learning via Separate Forward and Inverse Dynamics Pretraining reaches an average task length of 4.51 out of 5 on CALVIN ABC-D, against 4.33 for VPP. On SimplerEnv it reaches 51.2% success, against 27.7% for OpenVLA. It is the strongest competitor because it is the best video-to-action system in the file and it uses exactly the discrete bridge we are auditing. We first reproduce it. If our reimplementation falls more than a small margin below 4.51, we report results relative to our reproduction and state the gap. Beating it means a statistically significant gain in average task length over three or more seeds on CALVIN ABC-D, and a matching gain in SimplerEnv success, with the frozen tokenizer unchanged.

The decisive fairness control is a self-imitation baseline that receives the same environment rollout budget as our method. It keeps successful rollouts and adds them to behavior cloning, in the spirit of the self-supervised imitation in Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation, which reduced R2R's seen-unseen success gap from 30.7% to 11.7%. A second control is a codebook-usage fix in the style of UniCode: exponential-moving-average codebook updates with dead-code restarts. A third is behavior cloning with proprioception dropout, the robot analogue of word dropout from Bowman et al.. A fourth is InfoGAIL-style joint training, where the posterior and policy co-adapt without a frozen video vocabulary. A fifth is a continuous latent-action variant without quantization.

The ablations remove each reward term ($\lambda = 0$ gives pure per-token adversarial imitation; dropping $D_\psi$ gives pure identifiability) and swap mutual information for tie-aware $AP_T$. All methods share the same pretrained tokenizer and backbone. Each gets the same hyperparameter search budget of a fixed number of configurations, and every method that uses rollouts gets the same number of environment steps.

Why not just…?

Why not just fix codebook utilization? Usage fixes such as UniCode's synchronization make every token appear in the data. Identifiability is a different property: whether tokens remain distinct after the decoder acts. Our diagnostic reports both perplexity and $K_{\text{eff}}$ on the same checkpoints. If the usage-fixed baseline raises perplexity but leaves $K_{\text{eff}}$ and success flat, the distinction holds. If it raises $K_{\text{eff}}$ too, our repair is unnecessary, and we would report that.

Why not just drop proprioception during cloning so the decoder must read the token? This is the most serious threat. It targets the same shortcut, and it is cheap. It cannot, however, separate two tokens whose expert transitions look alike once the robot's own compounding errors are included, and heavy dropout may hurt fine control. We tune the dropout rate with the full search budget. If it matches our method on both $K_{\text{eff}}$ and CALVIN task length, the conclusion is that the problem was a decoder shortcut alone, and the adversarial component should be abandoned.

Why not just merge tied tokens into one? If ties were harmless redundancy, merging would cost nothing. Our claim is that ties hide distinctions the video model draws but the robot fails to execute. The test is to compare merged-codebook and repaired policies on the CALVIN subtasks whose expert frame pairs fall in tied classes. If merging matches the repair there, the claim is wrong.

Why not just use continuous latent actions or diffusion policies and avoid ties altogether? The file's evidence favors keeping discreteness. SAQ beats continuous offline methods by 2-3x on Robomimic, and on DECO-50 continuous Diffusion Policy reaches 51.25% against 72.25% to 82.50% for decoupled designs, per DECO. We still include a continuous latent-action variant so the comparison is direct rather than assumed.

Why not just collect more rollouts and self-imitate? This is covered by the matched-budget self-imitation baseline above. If filtered cloning on the same rollouts matches our method, the adversarial reward adds nothing beyond extra data.

Proposed Experiments

The main testbed is CALVIN ABC-D, which offers long-horizon chains where a single mis-executed token derails the remaining subtasks. SimplerEnv (Fractal) serves as a second simulator with DeFI and OpenVLA reference numbers. Robomimic Square provides a narrow-demonstration check against SAQ's roughly 44% success. In each environment we report the diagnostics ($K_{\text{eff}}$, tie mass, $AP_T$, token sensitivity) alongside success and task length.

Three checks are decisive. The first is that the untouched baseline shows substantial tie mass, meaning a meaningful fraction of expert transitions fall in tied classes at the largest codebook size. If tie mass is negligible, the premise fails and we stop. The second is that $K_{\text{eff}}$ predicts success better than perplexity, measured by rank correlation across checkpoints and codebook sizes. The third is that the repair beats both the reproduced DeFI decoder and the matched-budget self-imitation and proprioception-dropout baselines on CALVIN task length, and that the gains concentrate on subtasks whose tokens were tied before repair.

We also want to know whether identifiability transfers to a contact-rich setting. DECO-50 (dataset) provides real-hardware trajectories but no simulator. There we run only the offline diagnostic, using held-out trajectories as a proxy for rollouts, and report that no reported result found on Lacuna for latent-action identifiability on DECO-50 exists to compare against. We likewise found no reported result on Lacuna for codebook utilization or token identifiability in DeFI, so all diagnostic numbers will be our own measurements on our reproduction.