Some diffusion-style text generators are self-conditioned: at every denoising step the network sees both the current noisy input and its own guess of the clean output from the previous step. It then refines that guess instead of starting over. This feedback loop is what lets continuous language flows (generators that move a noisy embedding sequence toward clean text along a learned velocity field) compete with discrete and autoregressive models. The loop also makes these models awkward to compress. Today's hardware-friendly pruning methods for diffusion models calibrate on memoryless networks. They measure pruning damage one step at a time, on noisy inputs built from real data. "Hardware-friendly" here mostly means 2:4 sparsity: exactly two of every four consecutive weights are zero, a pattern NVIDIA sparse tensor cores execute at up to twice the math throughput. In a self-conditioned model, the error a pruned layer makes at one step comes back at the next step as part of the model's own input. That returning error can either shrink or grow along the trajectory, and no one has measured which.
We propose to prune self-conditioned flows in closed loop. We first measure how pruning error travels through the feedback loop at each step. We then choose 2:4 masks on activations collected from the sparse model's own self-conditioned rollouts, weighting each step by its measured closed-loop sensitivity rather than by a fixed heuristic. Finally, we recover quality with a short fine-tune that unrolls the few-step sampler, so the recycled guess comes from the sparse model's previous step. The target is a 2:4-sparse LangFlow that stays close to the dense model's 8-step generative perplexity while running measurably faster per step.
Core Claim
The hypothesis is that the self-conditioning loop acts as an error-correcting channel that current pruning calibration ignores. Masks chosen and repaired under the loop the model actually runs will beat masks chosen under the memoryless approximation at the same sparsity and the same recovery budget. The claim is falsifiable in both directions. If the loop amplifies pruning error, closed-loop sensitivity will be concentrated in a few steps and a small set of anchor-pathway weights, which should then be kept denser or wrapped. If the loop damps pruning error, uniform 2:4 sparsity should cost less on self-conditioned models than on memoryless ones.
Relation to Prior Work
The closest work compresses diffusion models under hardware constraints but assumes memoryless networks. CoDM combines step pruning, Taylor-expansion importance and 2:4 masks on Hopper sparse tensor cores for UNet image diffusion. OBS-Diff runs one-shot Optimal Brain Surgeon pruning, which removes weights using second-order (Hessian) estimates of the loss they cause. It supports 2:4 masks and weights early steps more heavily through a fixed logarithmic schedule. Diff-Pruning removes whole channels using only "informative" timesteps.
These three papers supply our method, since we reuse second-order 2:4 mask selection. They also frame a contradiction we must resolve. OBS-Diff says high-noise steps matter most, while Diff-Pruning discards them. None of them feeds a previous prediction back into the network. We replace their fixed timestep weights with weights measured through the feedback loop.
The self-conditioning line motivates the problem. Analog Bits introduced self-conditioning for discrete data. Directly Denoising Diffusion Models and Non-Cross Diffusion argue that the recycled guess corrects accumulated error and untangles crossing trajectories. LangFlow shows self-conditioning is what makes continuous embedding flows competitive. It reports perplexity 24.6 and generative perplexity 36.5 on OpenWebText. If the loop corrects error, pruning error is a natural test of that claim, and none of these papers performs it.
One paper constrains the design. Untied Self-Conditioning (USC) shows that in few-step sampling the solver folds the previous guess into the noisy state and then feeds the same guess again as the conditioning input. This redundant coupling collapses 8-step LangFlow to generative perplexity 531 on OpenWebText. USC's sampler, which damps that coupling, restores it to 62. Any pruning study that uses the stock sampler would therefore measure a coupling artifact rather than pruning damage, so we build on USC's sampler throughout.
On the hardware side, ARMOR wraps a 2:4 core in small block-diagonal matrices to recover accuracy. It also reports the price: 1.57x layer speedup versus 1.86x for a plain 2:4 layer in unfused PyTorch. We use ARMOR-style wrappers only where the diagnostic says the anchor pathway needs them.
Residual Context Diffusion recycles soft residuals in masked diffusion language models. It avoids memory blowup by detaching the recycled computation, and we adopt that trick in our unrolled fine-tune.
Research Program
The first stage is a diagnostic. For a dense LangFlow checkpoint running the USC 8-step sampler, we inject a controlled perturbation $\delta$ into the network's clean-embedding prediction at step $t$. We then record the closed-loop sensitivity $$s_t = \mathbb{E}\,\frac{\lVert \hat{x}_0^{\text{final}}(\delta \text{ at } t) - \hat{x}_0^{\text{final}} \rVert}{\lVert \delta \rVert},$$ where $\hat{x}_0^{\text{final}}$ is the final decoded embedding sequence. We measure $s_t$ under three conditions. In the first, the anchor input sees the perturbed guess. In the second, the anchor is frozen to the unperturbed guess. In the third, the perturbation enters only the anchor input. Comparing these conditions tells us whether the loop contracts or amplifies error, and at which steps. We then repeat the measurement with real 2:4 pruning error in place of synthetic $\delta$, layer by layer. This separates the anchor input projection, which is where USC located the redundant coupling, from the rest of the network.
The second stage is closed-loop calibration. We select 2:4 masks and compensating weight updates in OBS style, one layer at a time. Two things change relative to OBS-Diff: where the calibration activations come from, and how the steps are weighted. For each linear layer with dense weights $W$ we solve $$\min_{M \in \mathcal{M}_{2:4},\ \hat W} \sum_{t} w_t \,\big\lVert W X_t^{\text{cl}} - (\hat W \odot M) X_t^{\text{cl}} \big\rVert_F^2,$$ where $\mathcal{M}_{2:4}$ is the set of valid 2:4 masks, $\hat W$ holds the updated surviving weights, and $X_t^{\text{cl}}$ are layer inputs at step $t$. Crucially, $X_t^{\text{cl}}$ is collected from rollouts in which already-pruned earlier layers produced both the noisy state and the recycled guess. The step weights $w_t$ are set to the measured $s_t$. The calibration budget follows ARMOR's 128 samples, drawn from OpenWebText text.
The third stage is recovery. We run a short fine-tune that unrolls the sampler for $K$ steps with the sparse model generating its own anchor. To keep memory flat, we detach the graph between steps, as RCD does. This trains the loop to correct exactly the error the sparse network makes at inference.
Baselines
The naive reference point is magnitude 2:4. It keeps the two largest weights in each group of four across all of LangFlow, runs the USC 8-step sampler, and uses no calibration or fine-tuning. It sets the floor that ARMOR and OBS-Diff describe as severe degradation.
The strongest existing method is OBS-Diff in its 2:4 mode, applied to LangFlow and paired with the USC sampler. It uses the paper's logarithmic timestep weighting and calibration drawn from the dense model's forward-noised data. It is the strongest competitor because it is already second-order, already trajectory-aware and already supports 2:4. The only things it lacks are our closed-loop activations and our measured weights. Both methods get the same checkpoint, the same 128-sample calibration pool and the same USC sampler. When recovery is used, both get the same fine-tuning token budget and learning-rate sweep.
The dense USC model is the quality ceiling. USC reports generative perplexity 62 at 8 steps on OpenWebText, against 531 for the official sampler. No reported result was found on Lacuna for any 2:4-sparse self-conditioned language flow. The comparison numbers therefore have to be produced here.
Three ablations isolate the ingredients:
- Activations only: closed-loop activations with OBS-Diff's logarithmic weights, which isolates the effect of where calibration data comes from.
- Weights only: measured $s_t$ weights with open-loop activations.
- Recovery method: standard self-conditioned fine-tuning versus unrolled self-conditioned fine-tuning at an equal token budget.
The oracle-anchor control feeds the sparse network the dense model's recycled guess. The gap between this control and the full method measures how much damage travels through the feedback pathway specifically.
Beating the strongest baseline means the following. Our method must show lower 8-step generative perplexity than 2:4 OBS-Diff+USC at matched sparsity and budget, across three seeds. It must also keep sample unigram entropy no lower than the baseline's, since LangFlow already has a frequency bias that pruning could worsen.
Why not just…?
Why not just run OBS-Diff and then the USC sampler? This is the real threat, and it is our main baseline. OBS-Diff's weighting was derived for memoryless image models, where early error snowballs. In a self-conditioned flow, the recycled guess can either correct that error or re-inject it, and a fixed schedule cannot know which. If the diagnostic finds $s_t$ roughly proportional to OBS-Diff's logarithmic schedule, and the ablation shows no gain from closed-loop activations, then composing the two existing methods is sufficient. We would report that as the finding.
Why not just prune and fine-tune with the usual self-conditioning recipe? Standard training builds the recycled guess from the same noisy input at the same step. USC shows this differs from what the few-step sampler feeds the model, and the collapse from 62 to 531 is that mismatch at work. Pruning adds a second mismatch on top. The equal-budget recovery ablation tests directly whether unrolled training earns its extra forward passes.
Why not just take fewer steps instead of pruning? USC shows that 8 steps is already near the edge where self-conditioned flows break without special handling. Step reduction also leaves per-step cost, weight memory and bandwidth unchanged, while sparsity reduces all three. We compare the two directly at matched wall-clock. The comparison pits the sparse model at 8 steps against the dense model at the number of steps that fits the same latency, for example 6 steps. If the dense model with fewer steps wins, sparsity is not worth it for this model class.
Proposed Experiments
The primary testbed is LangFlow on OpenWebText, with LM1B as a second corpus, where LangFlow reports perplexity 30.0. We report three quality measures:
- generative perplexity at 4, 8 and 16 steps;
- validation perplexity;
- unigram entropy.
We report three speed measures:
- per-step and end-to-end latency using a native 2:4 sparse kernel on an Ampere or Hopper GPU;
- the fraction of the 2x theoretical speedup retained;
- USC's throughput retention (95.5% for the dense model).
For a modality check, we run the same procedure on self-conditioned Bit Diffusion for categorical CIFAR-10, where Analog Bits reports FID 6.93. This tests whether the effect is specific to language embeddings.
Three results would be decisive:
- Sign of the loop's effect. Real 2:4 error under the full loop should give smaller final error than the same error with the anchor frozen. If it does not, the error-correction premise fails, and the program falls back to protecting the anchor pathway.
- Closed-loop calibration beats open-loop. The activations-only ablation should beat OBS-Diff+USC before any fine-tuning. This shows the gain comes from where calibration data comes from, not from extra training.
- Real speedup. The fully sparse model must reach a real latency speedup above 1.5x per step. Otherwise any dense wrappers the diagnostic calls for are eating the benefit, as ARMOR's unfused 1.57x-versus-1.86x result warns.
References
- CoDM: A Co-design Framework for Efficient Sparse Diffusion Models
- OBS-Diff: Accurate Pruning For Diffusion Models in One-Shot
- Structural Pruning for Diffusion Models
- Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
- Directly Denoising Diffusion Models
- Non-Cross Diffusion for Semantic Consistency
- LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- Improving Few-Step Language Flows with Untied Self-Conditioning
- ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
- Residual Context Diffusion Language Models