TL;DR
This paper introduces CoDA, a 1.7B-parameter diffusion language model for code generation that achieves competitive performance with much larger models while maintaining efficient inference. By adapting a diffusion objective to a code-centric training pipeline and employing a novel progressive masking schedule, CoDA demonstrates the potential of smaller diffusion models for practical coding assistance.
What Problem Does This Paper Address?
Autoregressive (AR) language models have made significant strides in code generation, but they struggle with tasks requiring bidirectional context understanding, such as filling in missing code segments or editing large spans of text. Diffusion language models (DLMs) offer a promising alternative due to their inherent bidirectional context awareness and natural support for infilling. However, existing DLMs for code are often heavyweight, requiring substantial computational resources, which limits their practical deployment. This paper aims to bridge this gap by developing a more efficient and accessible diffusion-based coding assistant.
Methods
CoDA is built upon the Qwen3 backbone and adapted to a diffusion objective. The training process is divided into three stages:
- Pre-training: Large-scale diffusion pre-training on a diverse corpus of text and code (179.27B tokens) to equip the model with broad knowledge and reasoning capabilities.
- Mid-training: Code-centric mid-training (8.55B tokens) to further refine the model's understanding of programming languages and complex masking patterns.
- Post-training (Supervised Fine-Tuning): Instruction tuning on SFT subsets of the OpenCoder dataset to align the model with user intents for coding tasks.
A key innovation is the progressive masking schedule. This schedule gradually increases the difficulty of the training objective across epochs, employing three structured masking strategies:
- S1: Unmaskable Prefix: Forces the model to condition on the initial segment of the sequence, improving prompt reliance.
- S2: Truncated Suffix: Replaces a suffix with pad tokens, enhancing robustness to incomplete or noisy inputs.
- S3: Block Masking: Masks contiguous spans of tokens, simulating real-world infilling and editing scenarios.
To keep inference latency competitive, CoDA utilizes confidence-guided sampling, which reweights denoising updates based on per-token posterior entropy. The training infrastructure leverages Google's TPUs with PyTorch/XLA and a custom trainer for efficient distributed checkpointing and high-throughput data pipelines.
Datasets
Training Datasets:
- Pre-training:
dclm-baseline-1.0(Text, Web): 60.17B tokens- The Stack v2 (Python): 50.75B tokens
- The Stack v2 (train-smol): 38.22B tokens
- OpenWebMath (Text, Mathematics): 12.98B tokens
- RedPajama Arxiv (Text, Academic): 9.18B tokens
- Wikipedia (English, Text, Encyclopedia): 5.41B tokens
- DeepMind Mathematics (Text, Mathematics): 3.24B tokens
- Mid-training:
- RedPajama Arxiv (Text, Academic): 9.18B tokens
- Gutenberg (Text, ebooks): 3.24B tokens
- SmolLM-Corpus (python-edu, Code): 1.88B tokens
- OpenCoder Annealing Corpus (Code): 6.47B tokens
- The Stack v2 (Python, Code): 0.20B tokens
- Post-training: Stage 1 and Stage 2 SFT subsets of the OpenCoder dataset.
Evaluation Benchmarks:
- HumanEval
- MBPP
- EvalPlus (enhanced variants of HumanEval and MBPP)
The evaluation harness builds on existing frameworks like Dream and Qwen2.5-Coder, with adaptations for diffusion decoding.
Key Insights and Ideas
- Compact DLMs are Viable: CoDA demonstrates that diffusion models can achieve competitive performance even at a relatively small scale (1.7B parameters), challenging the notion that DLMs must be heavyweight.
- Progressive Masking is Crucial: The structured masking schedule is key to bridging the gap between training and inference, where masking strategies differ significantly. This approach improves robustness and adaptability.
- Instruction Tuning Boosts Performance: The CoDA-1.7B-Instruct model shows a significant improvement over the base model, highlighting the importance of fine-tuning for specific coding tasks.
- Diffusion Offers Complementary Biases: Even at modest scales, diffusion decoding provides inductive biases that complement AR models, leading to competitive performance on certain tasks.
- Inference Efficiency Trade-off: While more diffusion steps generally improve performance, gains saturate, suggesting that adaptive inference schedules can be employed for faster responses without sacrificing too much accuracy.
Key Results
- Competitive Performance: CoDA-1.7B-Instruct matches or surpasses diffusion models up to 7B parameters on HumanEval and EvalPlus benchmarks.
- HumanEval Improvement: CoDA-1.7B-Instruct achieves a 25-point improvement in pass@1 over CoDA-1.7B-Base on HumanEval, showcasing the impact of instruction tuning.
- Efficiency Gains: CoDA-1.7B-Instruct achieves 39.64% lower latency than Dream-7B-Instruct across comparable diffusion steps, attributed to its smaller parameter footprint.
- AR Model Comparison: CoDA-1.7B-Instruct trails the strongest AR baselines on HumanEval but delivers competitive MBPP performance, indicating a balanced profile.
Limitations and Future Work
Limitations:
- AR Baseline Performance: CoDA-1.7B-Instruct trails the strongest AR baselines on HumanEval.
- Model Scale: While competitive, the strongest diffusion baseline (Dream-7B-Instruct) still outperforms CoDA-1.7B-Instruct on MBPP-Instruct.
Future Work:
- Explore hybrid diffusion/AR decoding approaches.
- Investigate reinforcement learning-based fine-tuning.
- Develop more refined pre-training strategies to further enhance accuracy, inference speed, and applicability.
- The authors also plan to release model checkpoints, evaluation harnesses, and TPU training pipelines to facilitate community research.
Summary updated