Brain-Aligned Latent Actions: Imposing Premotor Cortex Geometry on a Robot's Action Vocabulary

Amit JadhavarXiv 2026 · arXiv

TL;DR

This paper demonstrates that the discrete "action vocabularies" learned by robot foundation models are highly unstable and underspecified by standard training objectives. The author leverages this flexibility to align a robot’s latent actions with the representational geometry of the macaque ventral premotor cortex (PMv), showing that neural priors can successfully shape how a model categorizes motion even for shapes it has never seen before.

What Problem Does This Paper Address?

Modern robot policies often use a "latent action vocabulary"—a set of discrete codes (tokens) that represent short bursts of motion. These are typically learned from unlabeled video by training a tokenizer to reconstruct the next frame.

However, the author reveals a major issue: these vocabularies are incredibly unstable. Two models trained with different random seeds or even different numerical precisions (fp32 vs. bf16) will assign the same robot motions to completely different codes, even if both models reconstruct video equally well. This "underspecification" means the training objective doesn't care which motion gets which code.

The paper asks: if the data leaves this choice open, can we use a prior from the biological brain to choose a better, more "natural" vocabulary?

Methods

The author uses Moto, an open-source latent-action model, and modifies its training using a dual-stream objective:

  1. Reconstruction Stream: The standard task of reconstructing robot video frames from the CALVIN dataset.
  2. Alignment Stream: The model encodes drawings of shapes made by a macaque monkey. A "straight-through" similarity loss pulls the geometry of the model's code usage toward the actual neural activity recorded in the monkey's ventral premotor cortex (PMv) while it drew those same shapes.

To measure how well the model matches the brain, the author uses Representational Similarity Analysis (RSA). This technique compares the "dissimilarity structure" of the brain (which shapes the brain thinks are similar) to the dissimilarity structure of the model's codes.

Figure 1: Brain-aligned latent actions. (A) Moto’s latent-action tokenizer encodes robot video into 128 discrete codes, trained as usual to reconstruct the next frame. It also encodes shapes drawn by a macaque (one typical recorded stroke per shape; dot: start), and a loss pulls the geometry of its code use toward the geometry recorded in ventral premotor cortex (PMv). Highlighted codes are illustrative. (B) On five held-out shape categories, only the PMv-trained tokenizer exceeds a kinematic yardstick (dashed), but an untrained descriptor, the stroke’s initial pen direction, scores higher (dotted; 0.65).

Datasets

  • Robot Video: The CALVIN dataset, featuring a robot arm performing manipulation tasks.
  • Neural Data: Electrophysiological recordings from the PMv of a macaque monkey drawing various shapes (circles, L-shapes, squiggles, etc.).
  • Held-out Shapes: Five shape categories (circles and four orientations of L-shapes) were kept entirely out of the alignment loss to test if the "brain-like" structure generalizes to new motions.

Key Insights and Ideas

  • The Freedom of Instability: The fact that latent vocabularies are unstable (Adjusted Rand Index of 0.001 between seeds) isn't just a bug—it's an opportunity to impose a prior without hurting the primary reconstruction task too much.
  • Neural Geometry Transfers: Aligning to the brain on a set of training shapes actually helps the model organize new, unseen shapes in a brain-like way. Non-neural targets (like language embeddings or random structures) failed to transfer this way.
  • Kinematics vs. Brain: A surprising "aha moment" was that a simple, hand-picked kinematic feature—the initial direction of the pen stroke—matched the PMv geometry better (RSA 0.65) than the trained model (RSA 0.52). This suggests that PMv's representation of these actions is heavily dominated by how a movement starts.

Key Results

  • Superior Alignment: The PMv-aligned tokenizer reached an RSA of 0.52 on held-out shapes, significantly higher than the off-the-shelf Moto tokenizer (0.20) or models trained with no prior (0.24–0.26).
  • The "Kinematic Yardstick": The model beat a "yardstick" of 100 random nonlinear kinematic encoders (95th percentile = 0.42), proving it captured structure beyond simple random motion features.
  • Reconstruction Cost: Alignment isn't free. The PMv-trained model had 1.76x higher reconstruction error than the baseline, as the model had to compromise between perfect video reconstruction and matching the neural geometry.
Figure 2: Premotor geometry transfers to held-out shapes. (A) RSA between each tokenizer’s code geometry and PMv on the five held-out shape categories, at the final checkpoint. Bars: mean over the four recording sessions of one training run; black lines: range over sessions; numbers: means (s.d. in Table 3). Vertical lines are untrained references. Dashed (0.42): the kinematic yardstick, each session’s 95th percentile of 100 random nonlinear encoders of pen motion (mean 0.26), averaged over sessions (Section A); only the PMv tokenizer exceeds it. The 95th percentile of the encoders’ four-session means is lower (0.39), and the shuffled tokenizer also exceeds that. Dotted (0.65): a hand-picked kinematic descriptor, the stroke’s initial direction, which exceeds every tokenizer. (B) The same score at every scored checkpoint (mean over sessions); PMv and shuffled PMv are emphasized, and the dashed line is the same yardstick. Single checkpoints are noisy because each session’s score is a rank correlation over ten distances, which is why the final checkpoint was fixed as the read-out in advance.

Limitations and Future Work

  • Not Beyond Kinematics: Because the "initial pen direction" baseline performed so well, the author notes they cannot yet claim the model has captured "neural" structure that goes beyond what simple kinematics could explain.
  • Single Animal/Small Set: The neural data comes from one monkey and a relatively small set of shapes.
  • The "So What?" Factor: While the paper successfully imposes brain-like geometry, it does not yet show if this actually makes the robot better at manipulation. The author has specified a downstream test on the CALVIN benchmark to answer this, but those results are pending.
  • Vocabulary Concentration: The PMv-aligned model used only 44 of its 128 available codes, suggesting the neural prior might be overly restrictive or "collapsing" the vocabulary.

Summary updated

Abstract

Robot policies increasingly act through a small vocabulary of discrete latent actions learned from unlabeled video, yet training leaves open which motion receives which code. We find this choice highly unstable: two independently trained runs of Moto's latent motion tokenizer assign the same robot clips to nearly unrelated codes (adjusted Rand index 0.001), and evaluating one tokenizer in bf16 instead of fp32 reassigns 74% of its tokens. We use this freedom to impose the representational geometry of macaque ventral premotor cortex (PMv), recorded while a monkey drew shapes, on the tokenizer's code assignment, with a straight-through similarity loss trained alongside its reconstruction objective on robot video. In one training run per target, on five shape categories the loss never saw, the PMv-trained tokenizer reaches a representational similarity of 0.52 with PMv, against 0.20-0.26 for released and no-prior tokenizers and 0.42 for the 95th percentile of random kinematic encoders. Kinematic, language, and random targets do not transfer (-0.17 to -0.03). Two results limit the claim: a hand-picked kinematic descriptor, the stroke's initial direction, matches PMv better (0.65), and a label-shuffled PMv control scores 0.41. We specify the downstream manipulation test, on a CALVIN pipeline that reproduces Moto's published results, before running it.
BibTeX
@misc{jadhav2026brain,
  author = {Amit Jadhav},
  title = {{Brain-Aligned Latent Actions: Imposing Premotor Cortex Geometry on a Robot's Action Vocabulary}},
  year = {2026},
  eprint = {2611.11674},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11674},
}
Figure