TL;DR
This paper demonstrates that the discrete "action vocabularies" learned by robot foundation models are highly unstable and underspecified by standard training objectives. The author leverages this flexibility to align a robot’s latent actions with the representational geometry of the macaque ventral premotor cortex (PMv), showing that neural priors can successfully shape how a model categorizes motion even for shapes it has never seen before.
What Problem Does This Paper Address?
Modern robot policies often use a "latent action vocabulary"—a set of discrete codes (tokens) that represent short bursts of motion. These are typically learned from unlabeled video by training a tokenizer to reconstruct the next frame.
However, the author reveals a major issue: these vocabularies are incredibly unstable. Two models trained with different random seeds or even different numerical precisions (fp32 vs. bf16) will assign the same robot motions to completely different codes, even if both models reconstruct video equally well. This "underspecification" means the training objective doesn't care which motion gets which code.
The paper asks: if the data leaves this choice open, can we use a prior from the biological brain to choose a better, more "natural" vocabulary?
Methods
The author uses Moto, an open-source latent-action model, and modifies its training using a dual-stream objective:
- Reconstruction Stream: The standard task of reconstructing robot video frames from the CALVIN dataset.
- Alignment Stream: The model encodes drawings of shapes made by a macaque monkey. A "straight-through" similarity loss pulls the geometry of the model's code usage toward the actual neural activity recorded in the monkey's ventral premotor cortex (PMv) while it drew those same shapes.
To measure how well the model matches the brain, the author uses Representational Similarity Analysis (RSA). This technique compares the "dissimilarity structure" of the brain (which shapes the brain thinks are similar) to the dissimilarity structure of the model's codes.
Datasets
- Robot Video: The CALVIN dataset, featuring a robot arm performing manipulation tasks.
- Neural Data: Electrophysiological recordings from the PMv of a macaque monkey drawing various shapes (circles, L-shapes, squiggles, etc.).
- Held-out Shapes: Five shape categories (circles and four orientations of L-shapes) were kept entirely out of the alignment loss to test if the "brain-like" structure generalizes to new motions.
Key Insights and Ideas
- The Freedom of Instability: The fact that latent vocabularies are unstable (Adjusted Rand Index of 0.001 between seeds) isn't just a bug—it's an opportunity to impose a prior without hurting the primary reconstruction task too much.
- Neural Geometry Transfers: Aligning to the brain on a set of training shapes actually helps the model organize new, unseen shapes in a brain-like way. Non-neural targets (like language embeddings or random structures) failed to transfer this way.
- Kinematics vs. Brain: A surprising "aha moment" was that a simple, hand-picked kinematic feature—the initial direction of the pen stroke—matched the PMv geometry better (RSA 0.65) than the trained model (RSA 0.52). This suggests that PMv's representation of these actions is heavily dominated by how a movement starts.
Key Results
- Superior Alignment: The PMv-aligned tokenizer reached an RSA of 0.52 on held-out shapes, significantly higher than the off-the-shelf Moto tokenizer (0.20) or models trained with no prior (0.24–0.26).
- The "Kinematic Yardstick": The model beat a "yardstick" of 100 random nonlinear kinematic encoders (95th percentile = 0.42), proving it captured structure beyond simple random motion features.
- Reconstruction Cost: Alignment isn't free. The PMv-trained model had 1.76x higher reconstruction error than the baseline, as the model had to compromise between perfect video reconstruction and matching the neural geometry.
Limitations and Future Work
- Not Beyond Kinematics: Because the "initial pen direction" baseline performed so well, the author notes they cannot yet claim the model has captured "neural" structure that goes beyond what simple kinematics could explain.
- Single Animal/Small Set: The neural data comes from one monkey and a relatively small set of shapes.
- The "So What?" Factor: While the paper successfully imposes brain-like geometry, it does not yet show if this actually makes the robot better at manipulation. The author has specified a downstream test on the CALVIN benchmark to answer this, but those results are pending.
- Vocabulary Concentration: The PMv-aligned model used only 44 of its 128 available codes, suggesting the neural prior might be overly restrictive or "collapsing" the vocabulary.
Summary updated