TL;DR
Categorical Flow Maps (CFMs) provide a promising continuous-state alternative to autoregressive language models, but their scalability to large-scale regimes has remained an open question. This paper successfully scales a CFM to 1.7B parameters trained on 2.1T tokens, demonstrating that self-distillation allows the model to generate high-quality text in as few as 4 inference steps. Additionally, the authors introduce a novel semi-discrete likelihood bound to evaluate these models on standard language benchmarks.
What Problem Does This Paper Address?
Autoregressive (AR) language models dominate modern NLP, but they generate text token-by-token, making inference inherently sequential and slow. Continuous diffusion and flow matching models have revolutionized image and audio generation by allowing parallel, few-step sampling and flexible steering (such as guidance and tilting), but adapting them to discrete text is challenging.
While "discrete diffusion" models exist, they operate on a rigid discrete grid, losing many of the mathematical advantages of continuous-state spaces. Categorical Flow Maps (CFMs) bridge this gap by modeling a continuous probability flow between a Gaussian distribution and one-hot encoded text data. However, CFMs require materializing massive logit matrices ($Context\ Length \times Vocabulary\ Size$), creating significant memory and computational overhead. Consequently, they had only been evaluated at modest scales (<1B parameters), leaving their viability for large language models (LLMs) unproven.
Methods
The authors implement a two-stage training paradigm to scale and optimize CFMs:
- Pretraining: They train a 1.7B-parameter continuous-state flow matching model using a cross-entropy loss to learn the probability flow from a Gaussian prior to the data distribution.
- Self-Distillation: To bypass expensive ODE integration during inference, they self-distill the pretrained model into a few-step generator using Progressive Self-Distillation (PSD).
To make this work at scale, they explore critical design choices: an error-decoding time schedule (which linearizes decoding errors over time), adaptive loss weighting to stabilize training, and random prefix unmasking to enable conditional generation (prompting). Crucially, they also derive a new semi-discrete Evidence Lower Bound (ELBO), allowing the model to compute text likelihoods for evaluation without expensive reverse ODE integration.
Datasets
- Training: The models are pretrained on the Nemotron pre-training dataset using OpenAI's TikToken tokenizer (vocabulary size $|V| \approx 100k$). Hyperparameter sweeps are conducted at a 400M parameter proxy scale over 16B tokens, while the final model scales up to 1.7B parameters over 2.1T tokens with a context length of 2048.
- Evaluation: Standard Multiple-Choice Question Answering (MCQA) benchmarks via the LM Harness, including ARC-E, BoolQ, OBQA, PIQA, RACE, and SIQA.
Key Insights and Ideas
- The Necessity of Prompting during Training: Unconditional training is insufficient for downstream utility. Without randomly unmasking prefixes ($p > 0$) during training to simulate prompts, the model completely fails at prompt completion tasks, causing distributional similarity scores (MAUVE*) to collapse to zero.
- Time Scheduling is a Balancing Act: Pure error-decoding schedules collapse training. A convex combination ($\lambda = 0.75$) of linear and error-decoding schedules provides the optimal trajectory for high-quality text generation.
- Divergence Matters in Distillation: While forward KL is standard, it can over-reinforce the imperfect teacher's biases. The authors found that Jensen-Shannon Divergence (JSD) bounds gradients and stabilizes training dynamics, though forward KL still holds a slight edge in final text quality.
- The Scaling Rule Conundrum: Standard zero-shot hyperparameter transfer rules (like $\mu$P or Complete(d)P) fail for CFMs because they do not account for the specialized continuous time-conditioning layers.
Key Results
- Competitive Perplexity: At 1024 inference steps, the undistilled 1.7B CFM achieves a generative perplexity of 58.66, approaching the autoregressive baseline of 45.33.
- High-Quality Few-Step Sampling: After self-distillation, the CFM generates high-quality text in just 4 steps, achieving a MAUVE* score of 0.88, which actually outperforms the autoregressive baseline (0.70).
- Non-Random Benchmark Performance: Using the newly derived semi-discrete ELBO, the CFM achieves non-random accuracy across MCQA benchmarks (e.g., 57.2% on PIQA, 47.1% on ARC-E), performing in a similar range to other non-autoregressive models like Eso-LM.
Method ARC-E BoolQ OBQA PIQA RACE SIQA
--------------------------------------------------------
Chance 24.7 50.4 26.6 51.6 24.2 32.2
AR (Ours) 73.1 63.6 40.2 76.9 36.3 43.5
CFM (Ours) 47.1 42.5 24.4 57.2 28.3 37.9
Limitations and Future Work
- Memory Bottleneck: Materializing the $L \times |V|$ logit matrix remains computationally heavy and scales linearly with context length and vocabulary size, requiring high-end hardware (e.g., 256 H100 GPUs) to manage.
- Hyperparameter Transfer: Developing zero-shot hyperparameter transfer rules specifically tailored for time-conditioned architectures remains an open problem.
- Likelihood Tightness: The derived semi-discrete ELBO bound is not perfectly tight, leaving a performance gap compared to discrete diffusion models optimized directly for ELBO maximization. Refining this likelihood estimation is a key direction for future work.
Summary updated