How Many Adapters Does a Diffusion Language Model Really Need Across Its Denoising Steps?

Diffusion language models write text by starting from a fully masked sequence and filling in tokens in parallel over a handful of passes, rather than one token at a time from left to right. Early passes see almost nothing and must lay out the global structure. Late passes see almost everything and only polish word choice. NaRA showed that fine-tuning such models with a fixed low-rank adapter (LoRA, a small trainable update $BA$ added to each frozen weight matrix) wastes this distinction. NaRA instead makes the adapter depend on the noise level, $\Delta W(\lambda) = B\,C(\lambda)\,A$. Here $\lambda$ is the fraction of masking, $A$ and $B$ are fixed low-rank matrices, and $C(\lambda)$ is a small $r\times r$ core matrix produced by a hypernetwork. This raises commonsense accuracy from 80.18% to 84.09%. The price is that the update changes at every step, so it can never be folded into the base weights, and every denoising pass pays for extra matrix multiplications.

This note proposes a measurement that doubles as a compression method. We sample the learned trajectory $C(\lambda)$ on a dense grid of noise levels and stack the results into a matrix. That matrix is fully observed, so we can apply the exact, closed-form Bayesian rank selection of Nakajima et al. to ask how many independent directions of variation the noise-dependence really has. The same tool, applied to $C(\lambda)$ at each fixed noise level, asks whether early planning steps use more adapter rank than late polishing steps. If the trajectory turns out to be nearly constant plus a rank-$K$ residual with small $K$, the constant part can be merged into the base weights. Only a tiny, analytically sized residual would stay live at inference. That would keep most of NaRA's accuracy at close to static-LoRA serving cost.

Core Claim

We claim that the noise-conditioning in trained noise-aware adapters is intrinsically low-dimensional along the noise axis. We further claim that analytic variational Bayesian matrix factorization (VBMF, a Bayesian low-rank decomposition that switches off unneeded components) selects that dimension without any tuning. Concretely, we split each layer's trajectory into a mean part $\bar C = \mathbb{E}_\lambda[C(\lambda)]$, which is merged into $W_0$, and a varying residual:

$$\Delta W(\lambda) \approx B\bar C A + \sum_{k=1}^{K} g_k(\lambda)\, B\, U_k A,$$

where $U_k$ are $r\times r$ basis matrices and $g_k(\lambda)$ are scalar coefficient curves, both read off the singular value decomposition. The number $K$ is chosen by VBMF's analytic threshold, which sets singular values below a noise-derived cutoff exactly to zero. Layers with $K=0$ become fully mergeable. Layers with small $K$ carry only $K$ rank-$r$ products gated by scalars. The claim is falsifiable in three ways. It fails if VBMF selects large $K$ in most layers. It fails if the compressed adapter loses most of NaRA's gain. It fails if a crude hand-tuned alternative matches it.

The new element is not "dynamic rank for diffusion adapters". NaRA's $C(\lambda)$ is an unrestricted $r\times r$ matrix, so it can already express a different effective rank at each noise level, and proposing "rank $r(\lambda)$" as a new architecture would be close to redundant. The new element is using analytic rank selection to measure and exploit the structure NaRA learns, so that we decide which parts of a noise-dependent adapter need to stay dynamic at all.

Why This Pairing Works Technically

The main barrier for analytic VBMF, noted in the file, is that the closed-form, column-decoupled solution holds only for fully observed matrices. It therefore breaks for recommender data and, naively, for masked token grids. Our matrix is not a masked token grid. Each row is $\mathrm{vec}(C(\lambda_j))$ for one grid point $\lambda_j$, and every entry is computed, so the setting is exactly the one Nakajima et al. solved. The solution comes from one SVD plus a scalar quartic per singular value. Their reported runtime is about 0.1 seconds against over 60 seconds for iterative coordinate ascent, which also got stuck in worse local minima. This makes an analysis per layer and per checkpoint cheap enough to run routinely. The companion result on Bayesian PCA with automatic dimensionality selection supplies the centered variant we need once $\bar C$ is subtracted. Alquier's survey, which shows that hierarchical-Gaussian-prior estimators reach near-optimal low-rank estimation rates, is the reason we trust the Bayesian threshold over an energy-fraction heuristic.

The threshold requires a noise variance. We estimate it from the spread of $C(\lambda)$ across independent training seeds, after aligning each seed's $A,B$ basis by orthogonal Procrustes. Jia et al. showed that masked diffusion models have intrinsically high training variance, so seed-to-seed variation is a real and sizeable noise floor, not a formality. In their framing, directions of $C(\lambda)$ that vary no more than seed noise are fit to gradient noise. The MDL rank-selection study constrains how we read the result. Squires et al. found that the selected rank grows with sample size, the "moustache effect". So a single $K$ is not an intrinsic constant. We report $K$ as a function of grid density and seed count and check that it saturates.

Research Program

The first stage reproduces NaRA on LLaDA-style diffusion LLMs, fine-tuned on Commonsense170k, Math14k, and a code set evaluated on HumanEval. We train five seeds per task at adapter ranks $r\in\{16,32\}$. Because NaRA uses a single global hypernetwork shared across layers, we analyze its output both as one shared trajectory and, where the architecture yields layer-specific cores, layer by layer.

The second stage is diagnostic. It answers the question NaRA left open: does early, heavily masked planning need more adapter rank than late refinement? For each $\lambda_j$ we run VBMF on the effective update $B\,C(\lambda_j)\,A$ and plot its selected rank against $\lambda$. For the whole trajectory we plot the spectrum of the stacked matrix and the selected $K$. We also inspect the coefficient curves $g_k(\lambda)$ to see whether they are smooth, monotone, or step-like. A step-like curve would mean the model has discovered a few discrete "noise regimes".

The third stage builds the deployable adapter. $\bar C$ is merged, the rank-$K$ residual is kept, and $g_k(\lambda)$ is stored as a tiny lookup table over the discrete step schedule, which removes the hypernetwork call. We evaluate this adapter with no retraining, and again after a short fine-tune of $U_k$ and $g_k$ with $A,B$ frozen. We also build a piecewise-constant variant that fits $g(\lambda)$ with $M$ bands by dynamic programming on the grid, and report how accuracy decays with $M$.

Baselines

The naive reference point is static LoRA at the same rank. NaRA reports 80.18% average across the 8-benchmark commonsense suite and 80.07% on Math14k. We retrain it under our exact data, step budget, and learning-rate sweep rather than quoting those numbers as our baseline.

The strongest existing method is full, unmerged NaRA. It reports 84.09% commonsense average, 80.89% on math, and is the only method above 40% on HumanEval. It is the ceiling on accuracy and the thing whose inference cost we are trying to remove. We run it from the same seeds that feed our analysis, with identical decoding schedules. Where early termination is used, we apply NaRA's block-wise early termination identically to all methods, because its reported 81% latency reduction is orthogonal to adapter cost. Beating NaRA means one of two outcomes. The first is recovering at least 80% of the NaRA-minus-LoRA accuracy gap while cutting per-step adapter compute and memory traffic to within 20% of static LoRA. The second is matching NaRA's accuracy outright with the adapter merged in most layers.

Band-wise LoRA is the direct competitor to the "few regimes" story. It trains $M$ separate static LoRAs, each on one uniform noise band, and merges each into its own weight copy or, for fair memory accounting, serves them as unmerged low-rank adapters. It receives the same total parameter and step budget, and $M$ is chosen by validation sweep. This sweep gives it an advantage our method does not get, because our $K$ is chosen analytically.

Two ablations isolate the key ingredient. The first replaces VBMF's analytic threshold with truncated SVD at a 95% energy cutoff, and with the best fixed $K$ found by oracle validation sweep. It tests whether Bayesian selection matters or whether any reasonable truncation works. The second keeps the full residual but merges only $\bar C$. It tests whether the mean/residual split alone captures the benefit.

Why not just…?

Why not just train NaRA at a smaller rank? A smaller $r$ shrinks the unmerged cost uniformly but does not make any layer mergeable, and it cuts the static and dynamic parts together. Our analysis predicts that most of the rank is needed for the static part and little for the dynamic part. We include NaRA at $r\in\{4,8\}$ in the sweep. If small-$r$ NaRA matches our compressed adapter at equal inference cost, the decomposition adds nothing and we will say so.

Why not just train a few band-specific LoRAs and skip the analysis? This is the most serious threat. If the true structure is two or three regimes, band-wise LoRA with a tuned $M$ may do as well. Our answer has two parts. First, VBMF says where the band boundaries and how many bands should be, and in which layers dynamics are needed at all. Band-wise LoRA must guess $M$ and apply it to every layer. Second, if band-wise LoRA with oracle $M$ equals our method, the contribution shrinks to the diagnostic finding plus a tuning-free choice of $M$. The head-to-head at matched budget settles this directly.

Why not use a learned rank allocator such as AutoLoRA? AutoLoRA tunes per-matrix LoRA ranks by bi-level meta-learning. It allocates a static rank per layer and has no notion of a noise axis, so it answers a different question. Its bi-level loop also costs full retraining, whereas our selection runs post hoc in seconds. We would run it only as a static reference if budget allows. It cannot, by construction, recover NaRA's noise-dependent gain.

Why not just accept NaRA's overhead? For single-request research use that may be reasonable. Diffusion LLMs, however, pay the adapter cost at every one of many denoising passes and cannot merge. The file lists this non-mergeability as an open limitation of NaRA. We measure actual per-step latency and memory traffic so the cost is quantified rather than assumed.

Proposed Experiments

The decisive experiment has three parts. First, a spectrum plot of the stacked $C(\lambda)$ trajectory for each task and layer, with VBMF's selected $K$ and a check that $K$ stabilizes as grid density and seed count grow. Second, accuracy on the commonsense suite, Math14k, and HumanEval for static LoRA, NaRA, band-wise LoRA, truncated-SVD variants, and our compressed adapter. Third, measured per-step latency, extra FLOPs, and memory footprint on the same hardware and decoding schedule.

The hypothesis is confirmed if three things hold. VBMF selects $K\le 2$ in most layers. The compressed adapter without retraining stays within one point of NaRA on the commonsense average. Truncated SVD at an energy cutoff either over-retains components, and so costs more, or under-retains them, and so loses accuracy, relative to the analytic threshold.

A secondary finding, reportable either way, is whether effective adapter rank rises or falls with the masking fraction. A positive answer would give the first direct evidence that global planning in parallel decoders consumes more adaptation capacity than local refinement. A flat answer would tell future adapter designs that noise-dependence should modulate scale, not capacity.

A final check probes generality. We repeat the analysis on adapters trained for a noise-conditioned translation setting on IWSLT16 En–De, if a diffusion translation model is available. This tests whether the low-$K$ structure is a property of denoising trajectories or of the specific reasoning tasks NaRA used.