TL;DR
Clover-35B-A3B-Preview is a 35-billion parameter Mixture-of-Experts (MoE) diffusion language model that achieves state-of-the-art performance on agentic tasks like coding and tool use. By combining recurrent memory for historical context with bidirectional attention for local "block" denoising, Clover bridges the gap between the parallel efficiency of diffusion and the high-reasoning capabilities of top-tier autoregressive models.
What Problem Does This Paper Address?
While standard autoregressive (AR) models dominate the LLM landscape, their sequential nature ($O(N)$ inference) is a fundamental bottleneck. Diffusion Language Models (DLMs) offer a path to parallel generation, but they have historically struggled with "agentic" capabilities—tasks requiring complex tool use, multi-step reasoning, and repository-level coding.
Existing DLMs often trail behind models like Qwen on benchmarks such as SWE-bench. Furthermore, scaling DLMs to handle long contexts efficiently remains a challenge. The DiffuSpace team introduces Clover to prove that a diffusion-based architecture can not only match but exceed AR performance in demanding agentic environments while maintaining high inference throughput.
Methods
Clover utilizes a hybrid backbone consisting of 40 decoder layers, alternating between Block-diffusion Gated DeltaNet (BD-GDN) layers and full-attention layers.
Key Innovations:
- BD-GDN Architecture: This component handles context in two ways. For the "clean" historical prefix, it uses a gated delta-rule recurrence (similar to Mamba2) to compress memory efficiently. For the "noisy" current block being generated, it enables bidirectional interaction, allowing tokens to attend to both past and future context within that block.
- Bidirectional Short Convolution: Unlike standard causal convolutions, Clover adds a zero-initialized "right-context" kernel. This allows the model to learn local bidirectional features during the denoising process without leaking information from future blocks.
- Block-aware Expert Routing (dMoE): To avoid the overhead of independent token routing in a parallel setting, Clover selects a "coreset" of experts for an entire block. Individual tokens are then routed within this shared pool, encouraging expert reuse and boosting throughput.
- Self-Speculative Decoding: Clover uses its own internal causal GDN to verify draft blocks. It employs an entropy-based acceptance rule to decide which tokens to commit, balancing speed and generation quality.
Datasets
The model was evaluated on a massive battery of 18 benchmarks, including:
- Agentic/Coding: SWE-bench Verified, Terminal-Bench 2.1, BFCL-v4 (Tool use), and SciCode.
- Reasoning/Math: MATH-500, AIME 2026, and ZebraLogic.
- General Knowledge: GPQA Diamond, MMLU-Pro, and HLE.
- Internal Workloads: An "agent replay" corpus of 64 historical trajectories and a "daily chat" set for long-form response testing.
Key Insights and Ideas
- Local Bidirectionality is a Superpower: The researchers found that prioritizing bidirectional interaction within the active block is crucial. It preserves the unique modeling strengths of diffusion (reconstructing sequences holistically) while the recurrent prefix keeps the "memory" cost low.
- Joint Decision Making: A fascinating variant of the model, the "Clover Decision Model," can predict up to eight related answers (e.g., multiple Sudoku cells or related legal judgments) in a single denoising pass. Because these predictions share intermediate features, the model achieves higher accuracy than if it answered the questions independently.
- Quantization-Friendly MoE: By using a finer group size (INT4 g32) for expert weights and optimizing expert placement between GPU and CPU, the team demonstrated that a 35B model can run effectively on consumer-grade hardware like an RTX 5070 Ti.
Key Results
- SOTA Agent Performance: Clover leads all evaluated models on agent metrics, scoring 73.6 on SWE-bench Verified (surpassing Qwen3.6's 65.8) and 51.7 on Terminal-Bench 2.1.
- Extreme Throughput: On an NVIDIA GB300, Clover reaches 680 tokens/s. Even on a consumer RTX 5070 Ti with 16GB VRAM, it achieves over 106 tokens/s using optimized expert placement.
- Decision Accuracy: In joint prediction mode, Sudoku cell accuracy jumped from 78.3% to 95.6%, and "all-correct" group accuracy on internal tasks rose from 65.4% to 77.0%.
- Latency Gains: Joint prediction provided a 7.58x speedup over sequential independent calls for multi-question tasks.
Limitations and Future Work
- Repetition: Like many diffusion models, Clover can occasionally fall into repetitive loops, requiring carefully tuned presence and repetition penalties.
- Drafting Overhead: On specific tasks like MATH-500, the computational cost of the drafting and verification phases can sometimes result in lower net throughput compared to highly optimized AR baselines.
- Future Directions: The team plans to extend Clover to multimodal inputs (vision-language-action), further increase context length, and refine the quality-latency trade-offs for quantized deployment.
Not discussed in the paper: Specific details on the pre-training data composition or the exact hardware used for the initial large-scale training runs.
Summary updated