In deep learning, we routinely face a choice between continuous and discrete representations. Continuous vectors possess an infinite capacity to capture fine-grained details, but they are highly susceptible to overfitting, domain-specific noise, and compounding errors in sequential generation. Discretization—mapping continuous features to a finite set of learnable vectors in a codebook—acts as a powerful regularizer. By forcing a network to communicate via a shared vocabulary of discrete symbols, we can filter out pixel-level noise, capture robust semantic concepts, and simplify downstream autoregressive modeling.
However, moving from continuous spaces to discrete codebooks introduces a fundamental tension: the codebook capacity tradeoff.
Small Codebook Size (Low Capacity) Large Codebook Size (High Capacity)
────────────────────────────────── ───────────────────────────────────
▲ Coarse abstractions ▲ High expressivity & detail
▲ High information loss ▲ Low representation bias
▼ Low variance & sample complexity ▼ High variance & sample complexity
▼ Easy downstream optimization ▼ Optimization instability / Collapse
If the codebook is too small, the model suffers from severe information loss, collapsing distinct real-world concepts into identical tokens. If the codebook is too large, the model faces optimization instability, codebook collapse, and an exponential explosion in the data required to generalize. Navigating this sweet spot is one of the central challenges in modern discrete representation learning.
Why Large Codebooks Fail: The Sample Complexity Penalty
Intuitively, it seems that making a codebook larger should always improve a model; a larger vocabulary simply provides more tools to describe the world. However, in a task-driven or reinforcement learning setting, an oversized vocabulary introduces a severe mathematical penalty.
In Task-Driven Discrete Representation Learning, Long Tung Vuong provides a theoretical foundation for this phenomenon, proving that the number of training samples required for generalization scales quadratically with the codebook size.
When a model maps continuous states to discrete codes, downstream components (like a policy network or a value function) must learn how to respond to each unique code. If the codebook size $K$ is unnecessarily large, the available training data is fragmented across too many distinct bins. While a massive codebook minimizes representation bias, it dramatically inflates the model's variance. In low-data regimes, this quadratic scaling law manifests as severe overfitting, making continuous representations paradoxically outperform discrete ones unless the vocabulary size is tightly constrained.
This sensitivity is echoed across diverse domains:
- Human Preference Alignment: In Latent Preference Coding, Gong et al. find that when decomposing human preferences into discrete latent factors, performance peaks at a moderate vocabulary size (32 to 64 codes). Exceeding this threshold causes the alignment algorithm to overfit to specific training prompts rather than learning generalizable human values.
- Offline Meta-Reinforcement Learning: Association for Artificial Intelligence 2026 et al. observe in Improving Generalization in Offline Meta-Reinforcement Learning via Cross-task Contexts that oversized codebooks cause an agent to overfit to specific training trajectories, destroying its ability to generalize to unseen tasks.
The Downstream Search Space Explosion
Beyond sample complexity, large codebooks cripple the networks tasked with predicting those codes. When using Vector Quantization (VQ) for action spaces or sequential generation, a downstream model (typically a Transformer) treats the codebook as a classification vocabulary.
In robotics, Lee et al. demonstrate this bottleneck in Behavior Generation with Latent Actions. If a robot's action space is quantized into an oversized codebook, the high-level policy faces a massive, complex search space. The categorical distribution over thousands of possible action primitives becomes incredibly difficult to optimize, leading to poor action selection.
Similarly, Xiaohan Lei notes in Primary-Fine Decoupling for Action Generation in Robotic Imitation that an oversized action vocabulary increases the difficulty of training high-level selection policies, causing models to fail at executing precise, multi-modal physical behaviors.
The Expressivity Floor: When Codebooks Are Too Small
If expanding the codebook breaks optimization, shrinking it too far introduces the opposite failure mode: under-expressivity. A tight communication bottleneck strips away domain noise, but it can also strip away critical structural and semantic information.
Continuous Input ──► [ Encoder ] ──► Continuous Latent (High Detail)
│
▼
VQ Bottleneck (Size K)
│
▼
Selected Codeword ◄─────┴─────► Information Loss
│ (Proportional to 1/K)
▼
[ Decoder ] ──► Coarse Reconstruction
When the codebook capacity is insufficient, the model experiences an expressivity floor:
- Visual Reconstruction: As highlighted by Wang et al. in Binary Latent Diffusion, small codebooks fail to capture fine-grained visual details, leading to blurry reconstructions or repeated artifacts. Ning et al. observe a similar capacity ceiling in All in Tokens when attempting to force complex visual task outputs like depth maps and segmentation masks into a small, unified token space.
- Motion and Gesture Synthesis: In DanceAnyWay, Aneesh Bhattacharya explains that a finite discrete vocabulary inherently limits movement diversity, making it nearly impossible to capture the subtle nuances and expressive details of continuous human motion. This tension between action space size and gestural expressiveness is a recurring theme in sequential motion modeling, as noted by Sun et al. in Co-speech Gesture Synthesis by Reinforcement Learning with Contrastive Pretrained Rewards.
Architectural Solutions to the Tradeoff
To break this deadlock, researchers have developed architectural strategies that decouple representation capacity from optimization complexity.
1. Fixed Grids Instead of Learned Codebooks
Traditional VQ-VAEs use online clustering (like Exponential Moving Averages) to update codebook vectors, which frequently suffers from "codebook collapse"—a state where only a fraction of the vocabulary is ever utilized.
To bypass this entirely, Mentzer et al. introduced Finite Scalar Quantization: VQ-VAE Made Simple. Instead of learning vector positions, FSQ projects continuous features onto a fixed, bounded scalar grid and rounds them to the nearest point. Because the grid is predefined, FSQ achieves 100% codebook utilization and scales gracefully to high capacities without the optimization instability or auxiliary losses that plague learned codebooks.
2. Coarse-to-Fine Decoupling
Rather than forcing a single codebook to handle both high-level semantics and low-level details, modern architectures split the responsibility.
Continuous Feature ──► [ Coarse Quantizer ] ──► High-Level Mode (Discrete Code)
│
▼
(Residual)
│
▼
[ Fine Policy ] ──► Low-Level Nuance (Continuous/Residual)
In Primary-Fine Decoupling for Action Generation in Robotic Imitation, Xiaohan Lei addresses the action-space tradeoff by using a two-stage framework. The model first selects a coarse discrete action "mode" (e.g., a high-level trajectory type) from a compact codebook, and then generates fine-grained continuous variations on top of it. Similarly, Lee et al. utilize Residual Vector Quantization (RVQ), which quantizes features hierarchically across multiple codebook layers, capturing fine details without exploding the primary vocabulary size.
3. Empirical Dimensionality Ratios
When designing standard VQ bottlenecks, the relationship between the codebook size ($K$) and the hidden dimensionality ($D$) of the vectors dictates information density. In SynergyNet, Gorade et al. provide an empirical rule of thumb for balancing this relationship: maintaining a codebook size approximately twice the hidden dimension ($K \approx 2D$) optimizes the balance between structural stability and representational capacity, preventing the latent space from becoming under- or over-saturated.
Key Foundational Work
Sequential Pattern Mining on Multimedia Data (2015) — contributed 1 concept elements to this direction.
Discrete-Valued Neural Communication (2021) — contributed 1 concept elements to this direction.