CoBoVLA: Decoupling Language from Action with Discrete Codes for Robust and Interpretable Robot Manipulation

Mehul Arora, Dhruv Agarwal, Ambuj AgrawalarXiv 2026 · arXiv

TL;DR

CoBoVLA (CodeBookVLA) is a modular 250M-parameter Vision-Language-Action model that decouples language understanding from robotic control using discrete, interpretable codebooks. This architecture achieves state-of-the-art robustness to paraphrased instructions and enables researchers to pinpoint exactly whether a failure was caused by a misunderstanding of words or a mistake in physical execution.

What Problem Does This Paper Address?

Current Vision-Language-Action (VLA) models are often "black boxes" that map raw text and images directly to motor commands. While they perform well on standard benchmarks, they are notoriously brittle. A simple paraphrase—changing "put the bowl on the cabinet" to "place the basin atop the storage unit"—can cause success rates to plummet from over 80% to as low as 8%.

Furthermore, when these end-to-end models fail, it is nearly impossible to diagnose why. Did the robot fail to understand the instruction, or did it understand perfectly but simply miss the grasp? This lack of interpretability, combined with the high computational cost of running large language models at every control step, makes deploying robust VLAs on real-world, resource-constrained hardware a significant challenge.

Figure 1. The same paraphrase and initial state, end-to-end versus through discrete codes. Top: an end-to-end VLA maps the words directly to actions; it fails, and nothing in the model shows why (frames from SmolVLA [33]). Bottom: CoBoVLA’s stage 1 maps the paraphrase to the task’s own step code, the planner turns it into a chain of action codes, and an actor executes it, and the task succeeds after 90 control steps (t).

Methods

The authors introduce CoBoVLA, which breaks the manipulation pipeline into three distinct, decoupled stages connected by discrete "codebooks":

  1. Stage 1: Step Coder. A small text encoder (38M parameters) maps the natural language instruction into a sequence of step codes. These codes represent the "meaning" of the task. Crucially, this stage is trained on paraphrase pairs so that different ways of saying the same thing map to the exact same discrete code.
  2. Stage 2: Visual Planner. A transformer-based planner (26M parameters) takes the current step code and visual observations to predict a chain of action codes. These are discrete motion primitives (e.g., "approach," "grasp," "lift") learned from demonstrations using a Minimum Description Length (MDL) segmentation process.
  3. Stage 3: Actor. A language-independent flow-matching transformer (99M parameters) converts the action codes into raw motor commands. Because the actor never sees the original text, it is completely invariant to how the instruction was phrased.
Figure 2. The CoBoVLA architecture showing the flow from instruction to discrete codes to actions.

Datasets

The model was evaluated on the LIBERO benchmark suite:

  • LIBERO-Goal: 10 manipulation tasks across 50 initial states.
  • LIBERO-Para: A diagnostic benchmark containing 4,092 unseen paraphrases of the LIBERO-Goal instructions.
  • Free-form Two-Step Tasks: A set of 160 episodes created by combining two single-step tasks into a single instruction (e.g., "Open the drawer and then move the bottle").

Key Insights and Ideas

  • Discrete Codes as a Robustness Bottleneck: By forcing language into a discrete "step code" interface, the model effectively filters out linguistic noise. If the text encoder maps a paraphrase to the correct canonical code, the rest of the system (planner and actor) is guaranteed to behave identically to the original instruction.
  • Traceable Failures: The discrete interface allows for "code-level failure analysis." Researchers can check the output of Stage 1; if the step code is wrong, the error is linguistic. If the code is right but the robot fails, the error lies in planning or execution.
  • Zero-Shot Composition: Because the model understands tasks as sequences of discrete steps, it can follow multi-step instructions (e.g., "Do A then B") even if it was never trained on multi-step demonstrations. It simply executes the code for step A, detects completion, and switches to the code for step B.
  • Efficiency through Decoupling: Unlike end-to-end models that must process the entire language instruction at every 20Hz control step, CoBoVLA encodes the text only once. The subsequent stages use compact discrete tokens, allowing the model to run in real-time on a standard laptop GPU.

Key Results

  • State-of-the-Art Robustness: On the LIBERO-Para benchmark, CoBoVLA achieved 81.1% success, vastly outperforming the 36.5% achieved by a SmolVLA baseline fine-tuned specifically on paraphrases.
  • Superior Base Performance: Even on standard instructions (LIBERO-Goal), CoBoVLA reached 91.4% success, compared to 82.8% for SmolVLA.
  • Diagnostic Power: Analysis showed that when instructions were correctly encoded into step codes, success reached 91%. When encoding failed, success dropped to 27.2%, identifying language understanding as the primary bottleneck for further improvement.
  • Real-Time Performance: The model runs with a latency of just 17ms per control step on an NVIDIA RTX 4050 laptop GPU, making it highly practical for edge deployment.

Limitations and Future Work

The authors acknowledge several limitations:

  • Simulation-Only: All experiments were conducted in the MuJoCo simulator; real-world deployment remains for future work.
  • Scripted Transitions: Multi-step execution currently relies on a scripted return to a "home" pose between steps to ensure the planner starts from a known distribution.
  • Instruction Diversity: LIBERO-Para only covers paraphrases of 10 underlying tasks. Scaling the meaning codebook to handle a much wider variety of open-ended instructions is a key next step.
  • Interference: In two-step tasks, the first action can sometimes physically interfere with the second (e.g., leaving a drawer open in a way that blocks a subsequent reach), which the current planner doesn't always anticipate.

Summary updated

Abstract

Vision-Language-Action (VLA) models achieve high success rates on the LIBERO benchmark but remain sensitive to paraphrasing. Existing VLAs ranging from 0.6B to 7.5B parameters experience a 22-52 point performance drop on LIBERO-Para while SmolVLA exhibits an even larger decline, from 83\% to 8\%. Because most VLAs jointly learn language understanding and action generation, it is difficult to determine whether a failure originates from misunderstanding an instruction or executing it incorrectly. To address these challenges, we introduce CoBoVLA, or CodeBookVLA, a 250M-parameter VLA that decouples language understanding, planning, and control through learned discrete interpretable codes. A text encoder maps the instructions to shared step codes, which a visual planner then converts into action code sequences executed by a language-independent actor. This design improves robustness to instruction paraphrasing while allowing failures to be traced to individual stages. It also enables efficient real-time control on resource-constrained hardware by encoding instructions only once and using compact discrete representations to keep each component lightweight. CoBoVLA achieves 91.4\% success on LIBERO-Goal, compared with 82.8\% for SmolVLA, under a matched protocol, with 17 ms per control step latency. On LIBERO-Para, it achieves 81.1\% across 4,092 unseen paraphrases and 78.5\% on matched episodes, compared with 36.5\% for SmolVLA LoRA fine-tuned on paraphrases. Without two-step demonstrations, CoBoVLA completes 48.8\% of 160 free-form two-step tasks. Finally, success reaches 91\% when instructions are correctly encoded but falls to 27.2\% otherwise, identifying language understanding as the main source of remaining failures. Together, these results demonstrate that decoupling language understanding and control can improve instruction robustness while making VLA failures easier to diagnose.
BibTeX
@misc{arora2026cobovla,
  author = {Mehul Arora and Dhruv Agarwal and Ambuj Agrawal},
  title = {{CoBoVLA: Decoupling Language from Action with Discrete Codes for Robust and Interpretable Robot Manipulation}},
  year = {2026},
  eprint = {2611.22395},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22395},
}
Figure