Abstract
Generating ultra-long text sequences (exceeding 10,000 tokens) remains a significant challenge for Large Language Models (LLMs) due to quadratic computational complexity and semantic feature dilution. We present DrDiff, a novel diffusion-based framework designed to break the efficiency-quality trade-off. DrDiff dynamically adjusts its architecture based on input length and complexity through three core mechanisms: Hierarchical Sparse Attention (HSA), Dynamic Expert Scheduling (DES), and Semantic Anchor States (SAS). Our results demonstrate that DrDiff, with only 220M active parameters, outperforms dense models as large as LLaMA-3.1-70B on long-context benchmarks while maintaining linear computational complexity and significantly reducing training overhead.
1. Introduction
The demand for ultra-long text generation—spanning technical documentation, long-form narratives, and comprehensive research summaries—has exposed two primary bottlenecks in current Transformer-based architectures. First, the standard self-attention mechanism scales quadratically ($O(n^2)$), making the processing of sequences beyond 10,000 tokens prohibitively expensive in terms of memory and compute. Second, feature dilution occurs as sequences grow; models often struggle to retain early-input semantic features, leading to logical incoherence and repetition.
Existing solutions typically rely on fixed sparse attention patterns or uniform denoising flows. However, these static approaches fail to account for the varying computational needs of different text segments or generation stages. DrDiff addresses these limitations by introducing a dynamic, multi-geared architecture that adapts to the specific requirements of the sequence length and semantic complexity.
2. Methodology
DrDiff integrates three dynamic technologies into the text diffusion process to optimize performance and efficiency.
2.1 Hierarchical Sparse Attention (HSA)
Rather than employing a singular attention pattern, HSA adaptively "shifts gears" based on the current sequence length. For short snippets ($\le 512$ tokens), the model utilizes standard dense attention. As length increases, it transitions to a combination of local and dilated patterns. For ultra-long sequences ($>8K$ tokens), DrDiff employs a dynamic global attention mechanism. This tiered approach ensures that computational complexity remains linear ($O(n)$) while preserving essential long-range dependencies.
2.2 Dynamic Expert Scheduling (DES)
DrDiff incorporates a Mixture-of-Experts (MoE) architecture within its diffusion denoising network. A specialized router allocates computational resources based on the complexity of the text segment. Repetitive or simpler segments are processed by "lighter" experts, whereas complex semantic junctions receive higher-capacity processing. This ensures that the model's "brain power" is distributed where it is most needed, preventing resource wastage on trivial tokens.
2.3 Semantic Anchor States (SAS)
To accelerate the typically slow sampling process of diffusion models, DrDiff utilizes Semantic Anchor States. These serve as "milestones" or core semantic summaries of the target text that guide the model toward specific intermediate states during denoising. By providing a guided trajectory, SAS allows for the use of efficient ODE solvers like DPM-solver++, significantly reducing the number of required sampling steps without sacrificing narrative coherence.
3. Experimental Setup
DrDiff was evaluated across a diverse suite of benchmarks designed to test long-context understanding and generation:
- LongBench: A comprehensive suite for QA, summarization, and few-shot learning.
- Multi-hop and Open QA: WikiHop, TriviaQA, and HotpotQA.
- Linguistic Tasks: OntoNotes (coreference) and Hyperpartisan News (classification).
- Summarization: Arxiv and Alpaca datasets.
- Stress Testing: Custom datasets from Project Gutenberg, PubMed Central, and Wikipedia were used to test limits up to 30,000 tokens.
4. Results and Analysis
4.1 Benchmark Performance
On the LongBench suite, DrDiff achieved an overall score of 33.5%. This performance surpassed several significantly larger models, including LLaMA-3.1-70B (32.1%), Longformer (31.0%), and the diffusion-based baseline DiffuSeq (29.5%). These results suggest that architectural specialization for long-form text can be more effective than raw parameter scaling.
4.2 Computational Efficiency
DrDiff demonstrated substantial efficiency gains, particularly at scale. For sequences exceeding 16K tokens, the model achieved a 56% reduction in training time compared to DiffuSeq and was approximately 9-10% faster than Longformer. The HSA mechanism successfully enabled the processing of 32K sequences on a single A100 GPU, a task where many competing models failed due to memory constraints.
4.3 Quality and Scaling
While most models experience a decline in quality as sequence length increases, DrDiff’s perplexity improved slightly during long-text generation (8K to 30K tokens). This indicates that the model effectively leverages additional context to improve its language modeling. However, a slight drop in BERTScore was observed as sequences approached the 30K threshold, suggesting a potential upper limit for semantic similarity retention.
5. Conclusion and Future Work
DrDiff demonstrates that dynamic architectural adjustments are key to breaking the efficiency-quality trade-off in long-text generation. By treating diffusion as a guided pathfinding problem and employing hierarchical attention, the model achieves state-of-the-art results with a fraction of the active parameters used by dense LLMs. Future research will focus on improving the interpretability of the MoE routing mechanism and exploring multi-modal extensions for scientific and legal document processing.
Summary updated