Action Discretization for Autoregressive Robotic Control

50 concepts42 papers2 proposals

Large language models and vision-language models excel because they treat the world as a sequence of discrete tokens. By predicting the next token in a sequence, they learn syntax, semantics, and complex reasoning. However, physical robots operate in a fundamentally different reality: a continuous space of joint velocities, forces, and spatial coordinates.

Historically, robotic imitation learning bridged this gap using continuous regression—predicting exact numerical values for actions by minimizing mean squared error. But continuous regression struggles with a core property of real-world behavior: multimodality. If a robot encounters an obstacle, it can choose to go left or go right. A regression model attempting to minimize average error will often split the difference, predicting an action right down the middle and crashing into the obstacle.

To solve this, a major research direction has emerged: action discretization. By packaging continuous physical movements into discrete tokens, researchers can frame robotic control as a next-token prediction task. This allows embodied agents to discard specialized regression heads, natively adopt standard Transformer architectures, and inherit the immense scaling properties of language models.


The Mechanics of Action Tokenization

How do we turn a continuous physical action—like moving a robotic arm or accelerating a vehicle—into a discrete word-like token? Current literature approaches this through two primary mechanisms: uniform binning and learned vector quantization.

Uniform Binning

The most straightforward approach is to split each continuous dimension of a robot's action space into a fixed number of equal-sized bins. For example, if a robot's gripper moves along the X-axis between -1.0 and +1.0 meters, this range can be divided into 256 discrete intervals, with each interval assigned a unique token identifier.

In [RT-1], Brohan et al. demonstrated that this binned coordinate system allows a Transformer to output actions sequentially (e.g., predicting X, then Y, then Z, then rotation). Expanding on this concept, Zitkovich et al. showed in [RT-2] that these action bins can be mapped directly onto existing entries in a large language model's text vocabulary. This allows a single unified network to process a visual scene, reason about a command, and output motor control commands using the exact same token-generation mechanism. Similarly, Huang et al. applied uniform binning to joint velocities in [LEO], enabling a decoder-only language model to act as a 3D embodied agent.

Vector Quantization (VQ)

While uniform binning works well for low-dimensional action spaces, it suffers from the curse of dimensionality when applied to complex, high-degree-of-freedom movements or long-horizon trajectories. If every dimension requires its own sequence of tokens, the context window quickly fills up.

To compress actions more efficiently, researchers employ Vector Quantization (VQ). Instead of tokenizing each dimension independently, an encoder maps an entire continuous action trajectory into a compact, discrete codebook of latent actions. As Vuong et al. note in [Action Tokenizer Matters in In-Context Imitation Learning], the way we package these movements into tokens is just as critical to model performance as the visual backbone itself; poor tokenization creates an artificial bottleneck that limits imitation learning.

To capture both macro-movements and fine-grained adjustments within a codebook, Lee et al. introduced [VQ-BeT], which utilizes Residual Vector Quantization (RVQ). RVQ tokenizes actions hierarchically: the first token captures the coarse, high-level trajectory of the movement, while subsequent "residual" tokens provide the high-resolution adjustments required for precise manipulation. Liu et al. leveraged this coarse-to-fine structure in [FASTer] to implement parallel block-wise decoding, significantly reducing the inference latency that typically plagues autoregressive models.


Why Discretization Works

Shifting from continuous regression to discrete tokenization fundamentally changes how a model learns behavior. The advantages materialize in two main areas: managing multimodal distributions and unlocking scaling laws.

Expressing Multimodal Distributions

When a model is trained on human demonstrations, it must learn to navigate choices. Because discrete tokenization frames action selection as a classification task, the model outputs a probability distribution (via a softmax layer) over the token vocabulary.

This allows the model to assign high probabilities to multiple distinct, valid trajectories simultaneously while assigning zero probability to the catastrophic "average" action. As highlighted by the authors of [RT-1], this capability is what allows tokenized controllers to robustly represent multiple valid paths for a single instruction, maintaining behavioral consistency where continuous models falter.

Unlocking Sequence Scaling Laws

Once actions are represented as sequences of tokens, robotic control becomes mathematically identical to language modeling. This allows reinforcement learning and imitation learning algorithms to ride the coattails of Transformer scaling laws.

For instance, Chebotar et al. developed [Q-Transformer], which applies conservative Q-learning autoregressively over tokenized action dimensions. This design allows high-capacity sequence architectures to scale effectively on multi-task robotic data, learning from thousands of failed autonomous trials alongside human demonstrations. Similarly, Ding et al. demonstrated in [QUAR-VLA] that discretizing control variables allows quadruped robots to optimize complex locomotion and whole-body manipulation using standard cross-entropy loss, directly benefiting from the sequence-modeling capabilities of pre-trained architectures.


Advanced Representations and Domain Adaptations

As the field matures, researchers are moving beyond basic tokenization schemes to address the computational overhead and unique spatial constraints of physical environments.

Bypassing Learned Codebooks

One drawback of vector-quantized tokenizers is that they require a separate, computationally expensive pre-training phase to learn the codebook before the main policy can be trained. To circumvent this, Zhou et al. introduced [BEAST]. Instead of learning a codebook from data, they mathematically encode continuous trajectories using B-splines and tokenize the spline coefficients. This approach guarantees smooth transitions between action sequences and eliminates the need for a separate tokenizer training phase entirely.

Domain-Specific Tokenization

In specialized applications like autonomous driving, tokenizing absolute spatial coordinates can cause generalization to fail, as a maneuver's coordinates change entirely depending on where it happens in the world.

To address this, Wang et al. explored domain-specific adaptations in [Do LLM Modules Generalize?]. They proposed an agent-centric tokenizer that discretizes acceleration space and kinematic states rather than global coordinates. By mapping intent directly to physical constraints, the model ensures that a specific action token represents a consistent physical maneuver (like a sharp left turn) regardless of the vehicle's global position on a map.

By transforming the continuous chaos of physical control into structured, discrete vocabularies, action discretization provides the vital translation layer needed to turn large sequence models into capable, multi-modal physical agents.

Key Foundational Work

RT-1: Robotics Transformer for Real-World Control at Scale (2023) — contributed 2 concept elements to this direction.
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023) — contributed 1 concept elements to this direction.
Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions (2023) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_action_discretization_f9785618,
  title = {action discretization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/action-discretization/rdr_f97856185ba64f9b942ca08ce6ca84ef},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_action_discretization_f9785618_v1,
  title = {action discretization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/action-discretization/rdr_f97856185ba64f9b942ca08ce6ca84ef/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake