Scaling Categorical Flow Maps

Oscar Davis, Anastasiia Filippova, Pierre Ablin, Victor Turrisi, Amitis Shidani, Marco Cuturi, Louis BéthunearXiv 2026 · arXiv

TL;DR

Categorical Flow Maps (CFMs) provide a promising continuous-state alternative to autoregressive language models, but their scalability to large-scale regimes has remained an open question. This paper successfully scales a CFM to 1.7B parameters trained on 2.1T tokens, demonstrating that self-distillation allows the model to generate high-quality text in as few as 4 inference steps. Additionally, the authors introduce a novel semi-discrete likelihood bound to evaluate these models on standard language benchmarks.

What Problem Does This Paper Address?

Autoregressive (AR) language models dominate modern NLP, but they generate text token-by-token, making inference inherently sequential and slow. Continuous diffusion and flow matching models have revolutionized image and audio generation by allowing parallel, few-step sampling and flexible steering (such as guidance and tilting), but adapting them to discrete text is challenging.

While "discrete diffusion" models exist, they operate on a rigid discrete grid, losing many of the mathematical advantages of continuous-state spaces. Categorical Flow Maps (CFMs) bridge this gap by modeling a continuous probability flow between a Gaussian distribution and one-hot encoded text data. However, CFMs require materializing massive logit matrices ($Context\ Length \times Vocabulary\ Size$), creating significant memory and computational overhead. Consequently, they had only been evaluated at modest scales (<1B parameters), leaving their viability for large language models (LLMs) unproven.

Methods

The authors implement a two-stage training paradigm to scale and optimize CFMs:

  1. Pretraining: They train a 1.7B-parameter continuous-state flow matching model using a cross-entropy loss to learn the probability flow from a Gaussian prior to the data distribution.
  2. Self-Distillation: To bypass expensive ODE integration during inference, they self-distill the pretrained model into a few-step generator using Progressive Self-Distillation (PSD).

To make this work at scale, they explore critical design choices: an error-decoding time schedule (which linearizes decoding errors over time), adaptive loss weighting to stabilize training, and random prefix unmasking to enable conditional generation (prompting). Crucially, they also derive a new semi-discrete Evidence Lower Bound (ELBO), allowing the model to compute text likelihoods for evaluation without expensive reverse ODE integration.

Datasets

  • Training: The models are pretrained on the Nemotron pre-training dataset using OpenAI's TikToken tokenizer (vocabulary size $|V| \approx 100k$). Hyperparameter sweeps are conducted at a 400M parameter proxy scale over 16B tokens, while the final model scales up to 1.7B parameters over 2.1T tokens with a context length of 2048.
  • Evaluation: Standard Multiple-Choice Question Answering (MCQA) benchmarks via the LM Harness, including ARC-E, BoolQ, OBQA, PIQA, RACE, and SIQA.

Key Insights and Ideas

  • The Necessity of Prompting during Training: Unconditional training is insufficient for downstream utility. Without randomly unmasking prefixes ($p > 0$) during training to simulate prompts, the model completely fails at prompt completion tasks, causing distributional similarity scores (MAUVE*) to collapse to zero.
  • Time Scheduling is a Balancing Act: Pure error-decoding schedules collapse training. A convex combination ($\lambda = 0.75$) of linear and error-decoding schedules provides the optimal trajectory for high-quality text generation.
  • Divergence Matters in Distillation: While forward KL is standard, it can over-reinforce the imperfect teacher's biases. The authors found that Jensen-Shannon Divergence (JSD) bounds gradients and stabilizes training dynamics, though forward KL still holds a slight edge in final text quality.
  • The Scaling Rule Conundrum: Standard zero-shot hyperparameter transfer rules (like $\mu$P or Complete(d)P) fail for CFMs because they do not account for the specialized continuous time-conditioning layers.

Key Results

  • Competitive Perplexity: At 1024 inference steps, the undistilled 1.7B CFM achieves a generative perplexity of 58.66, approaching the autoregressive baseline of 45.33.
  • High-Quality Few-Step Sampling: After self-distillation, the CFM generates high-quality text in just 4 steps, achieving a MAUVE* score of 0.88, which actually outperforms the autoregressive baseline (0.70).
  • Non-Random Benchmark Performance: Using the newly derived semi-discrete ELBO, the CFM achieves non-random accuracy across MCQA benchmarks (e.g., 57.2% on PIQA, 47.1% on ARC-E), performing in a similar range to other non-autoregressive models like Eso-LM.
Method      ARC-E   BoolQ   OBQA    PIQA    RACE    SIQA
--------------------------------------------------------
Chance      24.7    50.4    26.6    51.6    24.2    32.2
AR (Ours)   73.1    63.6    40.2    76.9    36.3    43.5
CFM (Ours)  47.1    42.5    24.4    57.2    28.3    37.9

Limitations and Future Work

  • Memory Bottleneck: Materializing the $L \times |V|$ logit matrix remains computationally heavy and scales linearly with context length and vocabulary size, requiring high-end hardware (e.g., 256 H100 GPUs) to manage.
  • Hyperparameter Transfer: Developing zero-shot hyperparameter transfer rules specifically tailored for time-conditioned architectures remains an open problem.
  • Likelihood Tightness: The derived semi-discrete ELBO bound is not perfectly tight, leaving a performance gap compared to discrete diffusion models optimized directly for ELBO maximization. Refining this likelihood estimation is a key direction for future work.

Summary updated

Abstract

Continuous diffusion and flow matching models could represent a powerful alternative to autoregressive approaches for language modelling (LM), as they unlock a host of advantages currently reserved for continuous modalities, including accelerated sampling and tilting. Recently, several works have demonstrated the possibility of generating discrete data continuously by a simple flow matching process between a Gaussian and the one-hot encoded data distribution. They have further shown the feasibility of accelerated sampling via Categorical Flow Maps (CFMs), resulting in competitive sample quality in the few-step regime. However, this method had only been evaluated at relatively modest scales ($<1$B), leaving the question of its scalability completely open. In this article, we train a $1.7$B-parameter base flow model on $2.1$T tokens and self-distill it into a CFM that generates diverse, high-quality text in as few as $4$ inference steps while maintaining near-data-level token entropy. Furthermore, we introduce a likelihood bound for CFMs in the semi-discrete setting, and show that they can be used to score the model on standard LM benchmarks, achieving results in the same range as discrete diffusion methods. Finally, we uncover some of the challenges that arise from training these models at scale, and we provide prescriptive insights on loss weighting and time scheduling.
BibTeX
@misc{davis2026scaling,
  author = {Oscar Davis and Anastasiia Filippova and Pierre Ablin and Victor Turrisi and Amitis Shidani and Marco Cuturi and Louis Béthune},
  title = {{Scaling Categorical Flow Maps}},
  year = {2026},
  eprint = {2605.07820},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.07820},
}
Figure