Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training

Tarun Suresh, Pranshu Chaturvedi, Hangoo Kang, Parth Shroff, Ishan S. Khare, Hermann Kumbong, Azalia MirhoseiniarXiv 2026 · arXiv

TL;DR

Context-Sharded Block Parallelism (CSBP) significantly accelerates long-context training for Block Diffusion Language Models (BDLMs) by exploiting the mathematical separability of the training objective. By keeping block-specific corrupted tokens local and sharding only the shared clean context across GPUs, CSBP reduces attention communication traffic by up to 93.5%, delivering up to a 7.59× throughput increase without altering training semantics.

What Problem Does This Paper Address?

Block Diffusion Language Models (BDLMs) are an emerging hybrid architecture designed for efficient inference. They maintain standard autoregressive (causal) dependencies across large blocks of text while generating the tokens within each block in parallel via diffusion denoising.

However, training BDLMs on long contexts (e.g., for coding agents or long-form assistants) hits a massive wall due to memory and communication bottlenecks. Existing distributed training systems rely on Context Parallelism (CP), which shards sequences across GPUs by token position. During the attention mechanism, standard CP blindly communicates both clean prefix tokens and corrupted (noisy) target tokens across all GPUs. Because corrupted tokens in one block never need to attend to corrupted tokens in another block, this symmetric treatment results in massive, highly wasteful cross-GPU communication overhead.

Methods

The authors introduce Context-Sharded Block Parallelism (CSBP) to eliminate this unnecessary communication through three main innovations:

  1. Block Parallelism (BP): Recognizing that the BDLM loss function naturally separates over target blocks, BP assigns the entire computation of a corrupted target block to a single GPU rank. This keeps all corrupted queries, keys, values, and gradients completely local.
  2. Context Sharding: While pure BP stops corrupted token communication, it forces GPUs to replicate long, overlapping clean prefixes, leading to Out-Of-Memory (OOM) errors. CSBP fixes this by token-sharding the shared clean sequence across the same ranks. Consequently, only the shared clean K/V pairs cross ranks during distributed attention.
  3. Dual-End Load Balancing: Because later blocks require more attention work due to longer clean prefixes, CSBP uses a dual-end strategy that pairs early and late blocks (e.g., block $b$ and $B+1-b$) on the same rank, perfectly balancing the computational load across GPUs.
Overview of training acceleration across different workloads and context lengths

Datasets

The authors evaluated their approach using the following datasets and benchmarks:

  • CoderForge-Preview: A dataset of successful programming agent trajectories formatted for the OpenHands interface, used for supervised fine-tuning (SFT).
  • LiteCoder-Terminal-SFT: Quality-filtered terminal interaction trajectories from the Terminus-2 environment.
  • Downstream Benchmarks: To measure real-world performance under fixed training budgets, models were evaluated on SWE-bench Verified (500 software engineering tasks) and Terminal-Bench Lite (100 terminal agent tasks).

Key Insights and Ideas

  • Objective Separability as a Superpower: The core "aha" moment is realizing that corrupted tokens only care about their own block and the clean past; they never interact with other corrupted blocks. Traditional parallelism strategies fail because they treat clean and corrupted tokens symmetrically.
  • Exploiting Architectural Quirks: For models like DiffusionGemma that utilize sliding-window attention, CSBP allows window-aware exchanges that send only the exact clean rows needed. This cuts logical attention traffic by a staggering 93.5%.
  • Decoupling Memory from Communication: Pure Block Parallelism reduces communication but breaks down due to memory replication. Sharding the clean context allows CSBP to achieve the communication savings of block isolation while maintaining the memory scaling of context parallelism.

Key Results

CSBP demonstrates dramatic throughput improvements that scale exceptionally well with context length:

  • SFT & Conversion Speedups: On 16 NVIDIA H200 GPUs at 256K context, CSBP improves throughput by 1.18–1.45× for SFT and 1.27–1.33× for converting standard autoregressive models to BDLMs. At 512K context, full-model speedup reaches 1.61×.
  • Speculative Decoding Breakthrough: When training DFlash2 speculative-decoding drafters on 8 H100 GPUs, CSBP achieves a 2.48× speedup at 512K and a massive 7.59× speedup at 1M context.
  • Wall-Clock Efficiency: In a fixed 12-hour training window, DiffusionGemma trained with CSBP achieved higher pass rates at every single intermediate checkpoint on both SWE-bench Verified and Terminal-Bench Lite compared to the best baseline.
Downstream performance comparison over a fixed 12-hour training budget

Limitations and Future Work

  • Training-Only Acceleration: CSBP accelerates the training phase because target tokens are known ahead of time. It does not speed up the generation/inference phase where future tokens are unknown.
  • Long-Context Dependent: The benefits of CSBP are highly pronounced in long-context regimes (64K to 1M+ tokens). For short contexts, traditional data parallelism or standard CP remains highly competitive or faster due to lower sequence-parallel overhead.

Summary updated

Abstract

Block diffusion language models (BDLMs) combine autoregressive dependencies across blocks with parallel denoising within blocks, but long-context training is constrained by distributed attention communication and activation memory. Conventional context parallelism (CP) shards the combined clean-plus-corrupted sequence by position, communicating shared clean K/V together with block-specific corrupted K/V and their gradients. We observe that the BDLM objective separates over target blocks. We introduce block parallelism (BP), a new distributed parallelism dimension that assigns each corrupted-block computation to one rank. To scale BP to long contexts, we introduce context-sharded block parallelism (CSBP), which also shards the shared clean sequence across those ranks. CSBP keeps corrupted K/V and gradients local, avoids replicated clean prefixes, and preserves BDLM training semantics. On 16 H200 GPUs at 256K context, CSBP improves throughput over the best baseline by 1.18-1.45x for supervised fine-tuning and 1.27-1.33x for conversion of autoregressive models to BDLMs, while matching or reducing peak HBM. Full-model speedup reaches 1.61x at 512K. On eight H100 GPUs, CSBP accelerates DFlash2 speculative-decoder training by 2.48x at 512K and 7.59x at 1M. In matched 12-hour DiffusionGemma 26B-A4B SFT runs, CSBP achieves higher pass rates at every trained checkpoint on SWE-bench Verified and Terminal-Bench Lite. Code: https://github.com/ScalingIntelligence/Turbo-dLLM
BibTeX
@misc{suresh2026block,
  author = {Tarun Suresh and Pranshu Chaturvedi and Hangoo Kang and Parth Shroff and Ishan S. Khare and Hermann Kumbong and Azalia Mirhoseini},
  title = {{Block Parallelism For Efficient Distributed Long-Context Diffusion Language Model Training}},
  year = {2026},
  eprint = {2609.19242},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.19242},
}
Figure