LAD: LoRA-Adapted Diffusion

Ruurd Jan Anthonius Kuiper, Lars de Groot, Bram van Es, Maarten van Smeden, Ayoub BagheriEMNLP 2025

TL;DR

This paper introduces LAD (LoRA-Adapted Diffusion), a novel framework that adapts existing autoregressive Large Language Models (LLMs) for non-autoregressive text generation using diffusion models. By leveraging Low-Rank Adaptation (LoRA) and a unique structured noising strategy, LAD achieves competitive performance with significantly reduced training costs compared to training diffusion models from scratch.

What Problem Does This Paper Address?

Autoregressive (AR) models, while dominant in text generation, suffer from inherent left-to-right decoding limitations, hindering efficiency and bidirectional reasoning. Diffusion models offer a promising alternative, excelling in continuous domains and showing potential for parallel generation. However, adapting diffusion models to the discrete nature of text efficiently and effectively, especially when leveraging large pre-trained AR models, remains a significant challenge. Training large diffusion models from scratch is computationally prohibitive, and existing methods for adapting AR models often have limitations in their ability to perform holistic sequence refinement or recover from early errors. This paper aims to bridge this gap by proposing a parameter-efficient method to adapt AR LLMs for diffusion-based text generation.

Methods

The core of LAD lies in adapting pre-trained AR LLMs (specifically Llama 3.2 and Llama 3.1 variants) for diffusion-based generation. This is achieved through two key innovations:

  • LoRA Adaptation: Instead of full fine-tuning, LAD employs Low-Rank Adaptation (LoRA) to efficiently adapt the AR models. LoRA freezes the majority of the pre-trained weights and injects small, trainable low-rank matrices into specific layers (query and value projections in the attention mechanism). This significantly reduces the number of trainable parameters, making training more efficient and mitigating catastrophic forgetting.
  • Structured Noising: LAD introduces a novel "structured noising" strategy for training. Unlike traditional Masked Diffusion Models (MDMs) that primarily rely on masking tokens, LAD combines token masking with structural perturbations like token swaps, duplications, and span shifts. This hybrid approach aims to mimic common iterative generation errors, enabling the model to learn corrections and perform holistic sequence editing, even when not all tokens are masked. The training objective is a standard cross-entropy loss, directly optimizing for reconstructing the original sequence from a corrupted version.

For inference, LAD offers two modes: a scheduled denoising approach with intermediate re-masking (similar to traditional MDMs) and a self-refining approach that relies on the model's ability to correct sequences without explicit re-noising.

Datasets

The models were trained on a combination of instruction-following datasets. Specifically, they used:

  • General Question-Answering: tatsu-lab/alpaca, vicgalle/alpaca-gpt4, and a filtered subset of crumb/Clean-Instruct-3M.
  • Task-Specific Corpora: To combat catastrophic forgetting, specialized datasets were included:
  • Multiple Choice QA: MMLU, ARC-Easy, HellaSwag
    • Math: GSM8K, ORCA
    • Coding: OpenCoder

All datasets were filtered to ensure prompts and outputs were less than 896 characters, tokenized, and formatted using standard Llama instruction templates. For evaluation, five benchmarks were used: ARC-Easy, MMLU, ARC-Challenge, HellaSwag, and GSM8K.

Key Insights and Ideas

  • LoRA is Sufficient for Diffusion Adaptation: The paper demonstrates that LoRA fine-tuning alone, when combined with a suitable noising strategy, is sufficient to adapt powerful AR LLMs into competitive diffusion models. This is a significant finding for parameter-efficient adaptation.
  • Structured Noising Enhances Correction: The hybrid structured noising approach, incorporating swaps, duplications, and span shifts alongside masking, proves effective in training models that can correct a wider range of structural errors in generated text.
  • Training Efficiency is Paramount: LAD achieves remarkable training efficiency, using only 0.001% to 0.008% of the token counts used by comparable large-scale diffusion models or their AR counterparts. This opens up diffusion-based generation to a wider range of research and applications.
  • Inference Flexibility: The ability to choose between denoising with and without re-noising offers a unique trade-off between generation speed and output quality, allowing users to tailor the process to their needs.

Key Results

  • Training Time: LAD models (1B, 3B, 8B) were trained in approximately 3, 8, and 16 hours, respectively, a fraction of the time and compute required for training from scratch.
  • Benchmark Performance: LAD models achieved competitive accuracy on most benchmarks (ARC-Easy, MMLU, ARC-Challenge, HellaSwag), often comparable to their base AR models, especially for similarly sized variants. Performance on GSM8K (complex reasoning) lagged behind the foundation models.
  • Fluency Metrics: Perplexity generally decreased with increasing model size and LoRA rank. Distinct 2-gram fractions remained high across models, indicating stable lexical diversity.
  • Efficiency: The parameter-efficient nature of LoRA adaptation led to significantly reduced training costs.

Limitations and Future Work

  • Model Size and Complexity: While LAD demonstrates scalability with model size, larger models (beyond 8B) might require more substantial GPU resources. The paper notes that the structured noising is hand-engineered and doesn't necessarily reflect intermediate inference samples.
  • Bidirectional Attention: The paper acknowledges that the explicit study of bidirectional attention and editing capabilities, a key feature of diffusion models, was beyond the scope of this work.
  • Evaluation Scope: The evaluation was limited to five benchmarks and two intrinsic metrics. Future work should expand to a broader range of tasks to better assess strengths and limitations, particularly in complex reasoning.
  • Diffusion Without Renoisng: While promising for speed, diffusion without renoising does not yet outperform diffusion with renoising in terms of output quality. Further evaluation is needed in this area.
  • Hyperparameter Tuning: The interactive interfaces allow for tuning, but a systematic study of optimal settings for hyperparameters like top-k, sharpness, and re-noising schedule is suggested for future work.

Summary updated

BibTeX
@inproceedings{kuiper2025lad,
  author = {Ruurd Jan Anthonius Kuiper and Lars de Groot and Bram van Es and Maarten van Smeden and Ayoub Bagheri},
  title = {{LAD: LoRA-Adapted Diffusion}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure