The most accurate hardware-friendly way to compress a large language model is currently to split each weight matrix into two parts. One part is a sparse matrix in the 2:4 pattern, meaning exactly two of every four consecutive weights are nonzero, which NVIDIA GPUs can multiply at up to twice dense speed. The other part is a small low-rank correction that repairs what the rigid sparsity pattern destroys. The best solver for this split, 3BASiL, uses a three-block ADMM. ADMM (alternating direction method of multipliers) is an optimizer that cuts a hard problem into pieces, solves them one at a time, and uses a "price" variable to push the pieces toward agreement. Like its predecessor HASSLE-free, 3BASiL keeps every weight in 16-bit floating point. The obvious next step is to quantize, storing the sparse values and the low-rank factors in 4 bits or fewer. Quantization adds more constrained pieces to the ADMM. That moves the solver into the regime where ADMM with three or more blocks is known to be able to diverge, and where every block is constrained to a non-convex set. No existing convergence result covers that case.
This note proposes to build and analyze a five-block ADMM for jointly sparse, low-rank, and quantized compression, and to make it stable with one specific mechanism. Our hypothesis is that, in this setting, multi-block instability shows up as assignment flicker: mask positions and quantization levels keep flipping between iterations because each block responds to stale versions of the others. We will measure flicker directly, tie the penalty schedule to it, and freeze assignments once they settle. After freezing, the remaining problem is continuous and can be analyzed with known multi-block results. The practical stake follows from an observation in the sparsity literature. Unstructured sparsity loses on hardware partly because its position indices cost 5 to 9 bits per nonzero, which exceeds the weight itself once weights are 3 or 4 bits (Non-structured DNN Weight Pruning Considered Harmful). In sparse-plus-low-rank compression, the 16-bit low-rank factor plays the same role: it is the overhead that dominates once everything else is quantized.
Core Claim
The claim has three parts. First, at a fixed total storage budget that counts every bit (values, 2:4 position metadata, low-rank factors, quantization scales), a jointly optimized sparse-plus-quantized-low-rank model beats two alternatives. One is the sequential pipeline: compress with 3BASiL, then quantize. The other is unstructured sparsity with quantization once its index bits are charged. Second, naively extending 3BASiL to five blocks produces measurable assignment flicker and reconstruction error that stalls or oscillates. Third, a penalty schedule driven by the measured flicker, followed by a freeze-and-polish phase, removes both problems.
The last part is the new ingredient. The rest of the solver is assembled from existing pieces: 3BASiL's split, the projection-onto-a-discrete-set step from ADMM-NN, and exact 2:4 projections from A Proximal Operator for Inducing 2:4-Sparsity.
The Problem and the Split
Each linear layer has dense weight $W$, calibration activations $X$, and damped second-moment matrix $H = X^\top X + \delta I$, where $\delta > 0$ is a small damping constant. We solve
$$\min_{S, U, V} \operatorname{tr}\big((W - S - UV^\top) H (W - S - UV^\top)^\top\big)$$
subject to two constraints. The sparse part $S$ must be 2:4 sparse with values on a 4-bit grid. The factors $U, V$ of rank $r$ must lie on a low-bit grid with per-group scales.
We introduce three copies. $D$ is the 2:4-and-quantized copy of $S$, and $Q_U$ and $Q_V$ are the quantized copies of $U$ and $V$. The coupling constraints are $S = D$, $U = Q_U$, $V = Q_V$. Over $S, U, V, D, Q_U, Q_V$ this gives five update steps per sweep: the continuous $S$, the joint low-rank update, and the three projections.
The continuous $S$ step is a linear solve with $H$. The low-rank step follows 3BASiL and uses randomized SVD. The $D$ step adapts the exact 2:4 proximal operator to a grid, and the factor copies are elementwise rounding. Every projected block is constrained to a non-convex set. That is precisely the case On the Global Linear Convergence of the ADMM with MultiBlock Variables leaves open, since its rank-condition scenarios need some blocks to be unconstrained. It is also the setting in which the counterexample reported in Iteration Complexity Analysis of Multi-block ADMM shows divergence is possible even for convex objectives.
3BASiL proves convergence for its own three-block scheme. We do not dispute that result. Our question is whether the guarantee and its practical behavior survive two more constrained blocks, and our expectation is that they do not without modification.
Research Program
The first deliverable is a diagnostic. At each sweep we record the fraction of 2:4 mask positions and grid levels that changed. This adapts the sparse-architecture-divergence measure used by SR-STE to describe mask instability in training. Alongside it we record primal residuals and layer reconstruction error. If our hypothesis is right, runs that stall show persistent flicker with residuals that oscillate rather than decay. Updating the price variables after every block instead of once per sweep should damp both. If flicker and stalling turn out to be uncorrelated, the hypothesis is wrong and we will report that.
The second deliverable is the stabilizer. Each block's penalty grows whenever that block's flicker rate exceeds a target. Because a large penalty pins a copy near its continuous partner, this forces the discrete assignments to stop changing in finitely many sweeps. Once flicker stays below a threshold for a window of sweeps, we freeze the mask and grid levels. We then polish the continuous values under the fixed structure. With the mask fixed, $S$ lives on a linear subspace and its objective is strongly convex because $H$ is damped. The polish therefore reduces to a two-block problem with a standard convergence guarantee.
The theory we will write down is limited and honest. In the frozen phase we prove linear convergence by checking the strong-convexity and rank conditions of Lin, Ma and Zhang. For the flicker phase we prove only finite-time freezing, not optimality. The open question we test empirically is whether freezing lands on good assignments.
The third deliverable is the storage accounting. Every comparison is reported at matched bits per original weight, roughly 3 and 4. The count charges unstructured methods the 5 to 9 index bits per nonzero reported by ADMM-NN and charges 2:4 methods their positional metadata exactly as the hardware format stores it. Under this accounting, the joint solver decides whether to spend bits on rank, on sparse precision, or on factor precision. That trade-off is invisible when the low-rank part is kept in 16 bits.
Baselines
The naive reference is the 2:4 model with no low-rank part, pruned by SparseGPT or Wanda and then quantized to the same budget. For orientation, A Proximal Operator for Inducing 2:4-Sparsity reports C4 perplexity of 43.30 for Wanda 2:4 on Llama-3.1 8B, against 26.44 for its proximal method, both unquantized.
The strongest competitor is 3BASiL followed by post-hoc quantization of $S$ and of $U, V$ with a Hessian-aware rounding step, at the same total bits. It is the strongest because 3BASiL is the best solver of the unquantized problem: it reports reducing the WikiText-2 perplexity gap to dense by more than 30% on Llama-3-8B at 2:4 plus rank 64. Quantization may simply commute with it. We run it with the same C4 calibration set (128 segments of 2,048 tokens), the same models, the same transformer-matching refinement, and the same wall-clock cap, and we tune its rank-versus-precision split by the same grid search. We have found no reported result on Lacuna for quantized sparse-plus-low-rank compression, so all numbers here will be our own. Our method beats this baseline only if it is lower in WikiText-2 and C4 perplexity, or higher in zero-shot average, at both budgets on at least two models, by more than seed variance.
A second competitor is ARMOR with its wrappers and 2:4 core quantized to the same budget. ARMOR reports roughly halving the 2:4 perplexity gap on Llama-2-13B using block-coordinate descent with no price variables. That makes it a natural test of whether dual mediation matters at all.
Two ablations isolate our ingredient. The first is the same five-block ADMM with a fixed penalty schedule and no freezing. The second is the same solver with a random block order each sweep, which is the cheapest generic remedy for ordering-induced lag. A third reference is unstructured pruning with quantization at matched bits including index cost, which tests the storage-accounting claim directly.
Why not just…?
Why not just quantize 3BASiL's output afterwards? This is the real threat. If post-hoc rounding of a good 16-bit solution loses little, the joint solver is unnecessary. The argument for joint optimization is that at 3 to 4 bits the best rank and the best sparse precision interact: a higher rank stored in fewer bits may beat a lower rank stored in more. A sequential pipeline cannot see that trade-off. The matched-budget comparison above settles the question. If the sequential pipeline ties us, we will report that the five-block machinery is not needed.
Why not just group everything into two blocks? One block would hold the continuous variables and the other all projections, and two-block ADMM converges more reliably. Two-block vs. Multi-block ADMM found the opposite in practice on multi-task learning: multi-block residuals were orders of magnitude smaller, and test error on the ADNI ADAS task was 5.09 versus 6.48 nMSE. That evidence comes from a convex problem, not compression, so we include the two-block grouping as an explicit baseline under the same budget and do not assume the result transfers.
Why not just keep unstructured sparsity and use fast sparse kernels? Sputnik beats dense cuBLAS only from about 71% sparsity, which is far above the 50% of 2:4. Its kernels also do not fuse transposes. Index bits make the storage case worse once weights are quantized. Our matched-bit unstructured baseline tests this directly instead of assuming it.
Why not just raise the penalty or run more iterations? A uniformly large penalty freezes assignments almost immediately, essentially at the initial rounding. More iterations do not remove oscillation that comes from block ordering. The fixed-schedule ablation, swept over several penalty values, shows whether flicker-adaptive control adds anything beyond tuning a single knob.
Proposed Experiments
We will compress Llama-2-7B, Llama-3-8B and Llama-2-13B using C4 calibration. We report WikiText-2 and C4 perplexity, the zero-shot suite (PIQA, ARC-E, ARC-C, HellaSwag, WinoGrande, RTE, OpenBookQA, BoolQ), compression wall-clock on one A100, and measured layer latency with 2:4 kernels. Latency is a secondary check; ARMOR reports 1.57x for its method against 1.86x for naive 2:4 on Qwen 2.5-14B.
Three checks are decisive. First, the five-block solver without the stabilizer should show higher flicker and higher final reconstruction error than with it, on the same layers. Second, the stabilized joint solver should beat post-hoc-quantized 3BASiL at matched bits. Third, the frontier from 3 to 4 bits should show the joint solver choosing a different rank-versus-precision split than the best sequential configuration.
If the first check fails, multi-block instability is not the bottleneck, and the contribution reduces to the storage accounting. If the second fails, the right recommendation is the simpler pipeline, and we will make it.
References
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
- A unified framework for Sparse plus Low-Rank Matrix Decomposition for LLMs
- Non-structured DNN Weight Pruning Considered Harmful
- ADMM-NN: An Algorithm-Hardware Co-Design Framework of DNNs Using Alternating Direction Method of Multipliers
- A Proximal Operator for Inducing 2:4-Sparsity
- On the Global Linear Convergence of the ADMM with MultiBlock Variables
- Iteration Complexity Analysis of Multi-block ADMM for a Family of Convex Minimization Without Strong Convexity
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
- Two-block vs. Multi-block ADMM: An empirical evaluation of convergence
- Sparse GPU Kernels for Deep Learning