Closed-Loop Delta Coding of Stale Context in Multi-GPU Diffusion

When a diffusion model generates a very large image on several GPUs, each GPU usually denoises one spatial patch. To keep the patches consistent with each other, every GPU still needs to see features from the rest of the image. The best current systems send these features one denoising step late, so the transfer can overlap with computation. Those transfers are sent at full 16-bit precision. They are also nearly redundant, because the features change very little from one step to the next. This proposal compresses them the way video codecs compress frames. Each GPU sends only a low-bit (2–4 bit) quantized change since its last transmission. Senders and receivers track the same reconstructed copy, so quantization errors cannot pile up over the denoising trajectory. The central claim is that coding the change, rather than the raw features, lets the cross-GPU context drop to roughly 3 bits per value with no measurable loss in image quality or seam quality. Quantizing the raw features at the same bit budget, even with the best outlier-smoothing tricks from the low-bit literature, should instead fall off the known sub-4-bit accuracy cliff.

This matters because the speedups from patch parallelism depend on the ratio of compute to communication. DistriFusion splits the image across GPUs and reuses features from the previous step, which it calls "stale" context. It gets a 3.0x speedup on 4 GPUs at an FID of about 24.3 against a single-GPU 24.0. The gain holds on fast interconnects. On slower links, such as PCIe or cloud nodes, the full-precision gather step becomes the bottleneck. Every existing patch-parallel system keeps that traffic in 16 bits: DistriFusion, PCPP, and PipeFusion.

Core Claim

DistriFusion already reports that adjacent denoising steps differ by an average of only 0.02. That is the property that justifies stale reuse. The same property makes the step-to-step change the natural thing to compress.

Let $A_t$ be the feature map a GPU contributes at step $t$, and let $\hat A_{t+1}$ be the reconstruction that both sender and receivers hold from the previous exchange. The sender transmits $Q(A_t - \hat A_{t+1})$, where $Q$ is a $b$-bit quantizer with per-tile scales. Both sides then update the shared copy:

$$\hat A_t = \hat A_{t+1} + Q(A_t - \hat A_{t+1}).$$

Because the residual is always taken against the shared reconstruction, the error after any step is just that step's quantization error. It does not accumulate. This is the classical closed-loop predictive coding argument. The error scales as $\mathrm{range}(A_t - \hat A_{t+1})/2^b$ instead of $\mathrm{range}(A_t)/2^b$. When the change is small relative to the feature range, this buys the equivalent of several extra bits.

The memory objection that sinks error-feedback schemes elsewhere does not apply here. TAH-QUANT quantizes activations sent between pipeline stages over slow networks. It rejects AQ-SGD-style error compensation because storing historical activations causes out-of-memory failures. In DistriFusion-style inference, however, every receiving GPU already stores the full stale context buffer. That buffer becomes the reference $\hat A_{t+1}$. Each sender needs one extra copy of only its own patch's reconstruction, which is roughly $1/n$ of the canvas when the image is split across $n$ GPUs.

Relation to Prior Work

Two lines of work motivate the proposal.

The patch-parallel systems establish the staleness regime, and all of them leave communication in 16 bits. DistriFusion gathers stale features from all GPUs. PCPP conditions each patch's attention only on a fraction of neighboring tokens and cuts traffic by about 70%. PipeFusion pipelines patches across layer stages and shards the attention key-value cache. Our method is orthogonal to how much context is sent: it compresses whatever context a scheme chooses to send, so it can be stacked on PCPP.

The distributed-quantization papers supply the method but target different parallelism. SDP4Bit quantizes weight differences to 4 bits in sharded data-parallel LLM training and reports a loss curve matching full precision. It is the closest precedent for difference coding. It works on weights across training iterations, not on spatial activations across denoising steps. It also does not need to keep a reconstruction shared across GPUs, because each iteration's weights are re-synchronized anyway. TAH-QUANT supplies the tile-wise group quantizer and the Hadamard-based outlier dispersal that we reuse on the residual.

Three single-GPU diffusion quantization papers constrain the design:

  • HadaNorm shows that Hadamard mixing on diffusion transformer activations fails unless channel means are removed first. Its reported quantization signal-to-noise ratio is –2.88 dB without its transform and 0.92 dB with it.
  • FP4DiT shows that activation distributions shift across tokens and timesteps and need token-wise online scales.
  • TR-DQ shows that outliers in diffusion models are timestep-dependent.

We expect the residual to be mean-centered almost by construction. This is a testable hypothesis. If it holds, the residual may be friendlier to rotation than raw activations, which would sidestep the failure HadaNorm identifies.

The research file raises two further concerns, and both look smaller than advertised. The first is that per-GPU dynamic scales would be "misaligned" at seams. That is not really a problem, because scales travel with each tile. The second is that channel rotations might "destroy spatial correlation" across halos, the overlapping border regions exchanged between neighboring GPUs. A per-token channel rotation leaves spatial layout untouched. We will verify both with seam-local metrics rather than assume them.

Research Program

We will build a drop-in communication layer for DistriFusion's asynchronous gather step and PipeFusion's patch transfers. The layer has three parts. The first is a closed-loop residual coder with tile-wise scales and a selectable bitwidth of 2, 3, 4, or 8. The second is an optional mean-centered Hadamard rotation of the residual. The third is a per-tile switch that sends an absolute refresh whenever the residual's energy approaches the raw feature energy.

The refresh switch is the honest answer to a known weakness. The file notes that stale-context methods break down on few-step distilled models, where features change substantially between steps. There, delta coding should lose its advantage. We want to measure where it stops helping rather than hide it.

The analysis has two parts. First, we will measure the residual statistics directly: range, channel means, outlier concentration, and how each varies with timestep, layer, and distance from patch borders. The aim is to predict, before running generation, the bitwidth at which each coder should fail. Second, we will separate the two error sources, staleness and quantization. We will do this by comparing against fresh-context and stale-full-precision references on identical seeds.

Baselines

The naive reference point is DistriFusion's "no communication" patching. On SDXL with COCO it reports a 3.9x speedup on 4 GPUs but an FID of 33.6. It marks the quality floor that any compressed scheme must stay far above.

The quality ceiling is full-precision DistriFusion at FID 24.3 against 24.0 on a single GPU, as reported in DistriFusion. We rerun it with the same model, prompts, seeds, step count, and hardware as every other arm. Beating it means matching its FID within seed noise, and its PSNR against single-GPU outputs, while sending 4–5x fewer bytes. It also means lower wall-clock latency on bandwidth-limited links.

The strongest competitor is absolute-value quantization of the stale context at the same bits per value. It uses TAH-QUANT's tile-wise groups plus HadaNorm-style mean-centered Hadamard rotation, both ported to this setting. This is exactly what a practitioner would try first, and it has published success at 3–4 bits in pipeline communication. We give it the same calibration data and scale-tuning budget as our coder. No result for quantized context communication in spatial patch parallelism was found on Lacuna, so the number for this baseline will be ours. We count it as beaten if our coder reaches equal FID at one or more fewer bits, or better FID and seam error at equal bits.

We also compare, at matched bytes per image, against PCPP's partial conditioning and against a "refresh every $k$ steps" scheme that simply skips transfers. PCPP reports 2.36–8.02x speedups on 4–8 GPUs versus 2.32–6.71x for DistriFusion.

Two ablations isolate the key ingredient. Open-loop delta coding, in the style of SDP4Bit, sends quantized differences of the true features without a shared reconstruction, so errors drift. Comparing it with our closed-loop coder tests whether the shared reconstruction is what makes low bits safe. We also switch the rotation and the refresh switch off one at a time.

Why not just…?

Why not just send the stale context in FP8 or INT8? At 8 bits this probably works, and we will say so plainly. The 8-bit absolute arm is included precisely to show whether the proposal adds anything there. The claim concerns the 2–4 bit regime. The quantization literature (Evaluating Quantized Large Language Models; Cui et al.) and HadaNorm's negative signal-to-noise results suggest absolute coding degrades sharply there. If absolute coding with rotation matches delta coding at 3 bits, the idea is refuted.

Why not just communicate less often, or send fewer tokens as PCPP does? Skipping a transfer is delta coding with the change quantized to zero. It is a special case our coder generalizes. PCPP reduces how many tokens are sent, while we reduce bits per token. The matched-bytes comparison settles whether bits or tokens are the cheaper thing to cut. The PCPP-plus-delta arm tests whether the two savings combine. PCPP reportedly fails on text and rigid geometry, so we report those prompt subsets separately.

Why not just use layer-wise pipeline or tensor parallelism with activation quantization, avoiding patches entirely? PipeFusion reports that tensor parallelism needs a synchronization at every layer. That is too slow for high-resolution generation. Dryden et al. show that layer pipelining cannot shrink the peak memory of a single very large activation, so spatial splitting is required at these resolutions. Our coder applies equally to PipeFusion's hybrid transfers, which we test directly.

Proposed Experiments

U-Net model. We run SDXL at 1024×1024 on COCO 2014 validation captions, following DistriFusion's protocol. We report FID against real images, FID and PSNR against single-GPU outputs, and a seam-band error. The seam-band error is computed only within a few latent pixels of patch borders, so seam tearing cannot hide inside a global average.

Diffusion transformer models. We run PixArt-Σ, the model evaluated by HadaNorm, and Flux.1 under PipeFusion, which it reports running on COCO 2014 captions. We report CLIP score and FID, and scale to 2048 and 4096 pixels.

Hardware. We use 4 and 8 GPUs on NVLink and on PCIe, and we throttle the link to cloud-like bandwidths. We report end-to-end latency, bytes per step, and peak memory, including the sender-side reconstruction buffer.

Stress test. We use a few-step distilled model, Flux.1-schnell at 1–4 steps, to locate where residual coding loses its edge and how often the refresh switch fires.

Decisive checks. The first check is the bitwidth at which each arm's FID and seam error depart from full-precision DistriFusion. We predict at least one bit of separation between closed-loop delta coding and rotated absolute coding. The second is the gap between closed-loop and open-loop coding. We predict it grows with step count, because open-loop errors drift. The third is whether the measured residual statistics predict these failure points. If they do, the result is a mechanism, not just a benchmark.