Linear attention models such as Gated DeltaNet avoid the quadratic cost of standard Transformers. A standard Transformer compares every token with every other token. A linear attention model instead keeps a fixed-size running memory: a matrix "state" per attention head that is updated once per token. Recent work shows this memory is mostly wasted. Its effective rank collapses, so half to three quarters of the key and query channels can be sliced off with little loss in perplexity. Because the slicing removes whole channels, the pruned model stays dense and runs faster on ordinary GPUs. What nobody has asked is how many joules these pruned models actually save, and on which workloads. Nor has anyone asked whether a pruning plan chosen to save energy looks different from one chosen by a uniform rule or by FLOP counts.
We propose to treat measured energy as the price of each removable structure in a linear attention model. The removable structures are recurrent-state channels, whole heads, and feed-forward neurons. The pruning plan is then allocated across layers and component types to buy the most quality per joule saved, for a stated serving regime: short prompts at large batch versus long-document decoding with prefix caching. The bet is that these regimes have different energy cost structures. Shrinking the state cuts memory traffic per decoded token and the size of cached prefix checkpoints. Shrinking feed-forward width cuts weight fetches. If that holds, the energy-optimal plan should change with the regime and beat the uniform plans used today on long-context benchmarks at equal measured energy.
Core Claim
The state-reduction papers, On State Reduction in Linear Attention and The Key to State Reduction in Linear Attention: A Rank-based Perspective, supply the method we reuse. They introduce Deep Rank Revealing QR (DRRQR), which uses QR decomposition with column pivoting on joint key and query statistics to choose which channels to keep. They then apply brief LoRA recovery fine-tuning. On Gated DeltaNet 1.3B, pruning half the channels moves Wikitext perplexity from 16.8 to 17.3 and gives a 1.28x H100 speedup.
These papers also motivate the gap in three ways. They use the same ratio for every head even though they observe that rank use varies widely across heads. They report no energy figures. They evaluate no long-document benchmarks, while noting that recall-heavy tasks are sensitive to state reduction.
The physical argument for why energy should be measured directly comes from EcoFormer. It notes that DRAM accesses cost orders of magnitude more energy than arithmetic, so FLOP reductions that do not reduce memory movement save little energy.
Our claim has three parts. First, during long-context decoding, energy per token scales with state bytes moved rather than with FLOPs. Second, an allocator that ranks every removable unit by quality loss per measured joule will find non-uniform plans that dominate uniform DRRQR on the quality–energy frontier. Third, the plan it finds for long-document decoding will differ materially from the plan it finds for short-prompt serving.
Research Program
We would build the allocator in three pieces.
Energy cost model. We profile each layer of Gated DeltaNet 1.3B and Mamba-2 on H100 with GPU energy counters, using the process-isolated accounting of Eco2AI. For each layer we record the marginal joules saved by removing one block of state channels, one head, or one block of feed-forward neurons. We do this separately for prefill and decode, at several batch sizes and context lengths up to 64k tokens. Channels are removed in groups that respect kernel alignment, because the fast depthwise convolution kernels require axis-aligned slicing. Each layer keeps a single head width, so existing kernels run unchanged.
Damage model. We score each unit by the increase in loss when it is removed. The loss is measured on a long-sequence calibration set, and only on tokens whose prediction depends on context beyond a few thousand positions. This targets the long-range recall that state pruning is known to hurt. As an inexpensive prior, we also use the DRRQR residual curve and the decay-derived retention horizons from DASC.
Allocation. For a regime $w$ with an energy budget $B$, we choose a set $P$ of removed units to solve
$$\min_{P} \sum_{u \in P} \Delta L(u) \quad \text{subject to} \quad E_w(\text{full}) - \sum_{u \in P} \Delta E_w(u) \le B,$$
where $\Delta L(u)$ is the calibration loss increase from removing unit $u$, and $\Delta E_w(u)$ is its measured energy saving under regime $w$. We solve this greedily by the ratio $\Delta L/\Delta E_w$. We then re-score after each tranche, because removals interact, and finish with the same LoRA recovery step as DRRQR so that recovery cost stays the same across methods.
This extends the layer-adaptive idea of LAST, which allocates state pruning across layers of S4/S5 models by $H_\infty$ scores normalized by output energy, in two ways. We price units in physical joules rather than signal energy, and we target modern linear-attention language models. It also carries the width-plus-depth coordination of 2SSP from quadratic Transformers into models where the attention block has its own state bottleneck. 2SSP uses perplexity alone and has no energy term.
Baselines
Dense model. The naive reference is the unpruned Gated DeltaNet 1.3B and Mamba-2, measured with identical energy instrumentation. This gives the full-quality, full-energy corner of every plot.
Uniform DRRQR. The strongest existing competitor is uniform DRRQR with recovery fine-tuning from On State Reduction in Linear Attention. At 50% compression it reports Wikitext perplexity 17.3 against 16.8 dense, an LM-eval average of 58.3 against 59.4, a 1.28x H100 speedup, and 0.73x VRAM. It is the right competitor because it already captures most of the rank redundancy with hardware-friendly slicing. We would rerun it ourselves at 50% and 75% with the same calibration data, the same LoRA recovery tokens and steps, and the same hardware, then measure its joules. To beat it, we must achieve higher LongBench and recall-task scores at equal or lower measured energy per query, with perplexity within noise.
DASC. For prefix-cached serving, the competitor is DASC. It leaves weights untouched and compresses cached state checkpoints by retention horizon. It reports 2.63x to 28x checkpoint compression and 42.6% lower mean time to first token. We would run it on the same models and request traces and compare energy per served request and cache capacity.
2SSP adaptation. We would port 2SSP to linear attention by treating whole attention blocks as depth units. On Phi-3 14B at 37.5% sparsity it reports a 58.92% zero-shot average; there is no reported result on Lacuna for linear-attention models.
Ablations. Two ablations isolate the key ingredient. The first uses the same allocator with FLOPs in place of measured joules. The second uses the same allocator with a single regime-agnostic cost. If either matches the full method, the energy pricing is not doing the work.
There is no reported result on Lacuna for any pruned linear-attention model on LongBench, and none for energy per query of DRRQR-pruned models. We will generate these numbers ourselves.
Why not just…?
Why not just run uniform DRRQR and attach an energy meter? This is the real threat. If energy savings simply track state size, uniform DRRQR plus measurement would close Gap 1 at almost no cost. Our position is that uniform ratios waste budget on high-rank heads and spare low-rank ones, which the source paper itself observes. The FLOP-priced ablation and the uniform baseline settle this directly. If the allocator's frontier does not separate from uniform DRRQR by more than seed variance, we would report the measurement study as the contribution and drop the allocator claim.
Why not just quantize the state and weights? Quantization shrinks bytes without changing shapes, and it is well supported by hardware. However, DASC found dimension reduction and INT8 state quantization to be complementary, reaching 8.11x combined capacity. Quantization also leaves the unused state rank in place. We would run INT8 state quantization alone and stacked on each pruning plan. The prediction is that stacking with our plan dominates both.
Why not just compress at serving time with DASC and leave weights alone? DASC never shrinks the prefill computation or the per-token state update. It also requires ragged layouts. Permanent pruning might remove the need for those layouts, but this is an open question, and DASC may win when long prompts are reused heavily. Our serving experiment measures both methods, and their combination, on the same request traces.
Why not just route easy queries to a small model, as EcoThink does? EcoThink saves 40.4% average energy through routing. However, it cannot reduce the cost of a query once that query takes the deep path. LongBench reports that retrieval-style truncation hurts tasks that need reasoning spread across a document. Routing is orthogonal to this work, and we restrict our claims to queries that require the full long context.
Proposed Experiments
The first check tests the physical premise. We measure decode energy per token against state bytes and against FLOPs across pruning levels and context lengths of 4k to 64k. The premise holds only if state bytes predict energy markedly better than FLOPs.
The main experiment compares quality–energy frontiers for the dense model, uniform DRRQR, the FLOP-priced allocator, and the energy-priced allocator. Quality is measured on LongBench by length bucket, on the recall suite (FDA, SWDE, SQuAD, NQ, TriviaQA, Drop), on GovReport-long summarization, and on Wikitext and Lambada perplexity. Energy is reported as joules per query. We report Carburacy from Carburacy with a fixed grid factor, so carbon conversion does not confound the comparison, and also the perplexity-energy product.
The decisive regime test optimizes one plan for short-prompt large-batch serving and one for long-document prefix-cached decoding, then evaluates each plan in the other regime. If the energy-priced view is correct, each plan should win in its own regime. If one plan wins both, regime-specific allocation is unnecessary and we would say so.
Finally, we report the one-time energy cost of profiling and recovery fine-tuning. We then compute the break-even number of served queries, since a pruning method that consumes more energy than it saves before deployment amortizes it fails the sustainability test it claims to pass.
References
- On State Reduction in Linear Attention
- The Key to State Reduction in Linear Attention: A Rank-based Perspective
- EcoFormer: Energy-Saving Attention with Linear Complexity
- Eco2AI: carbon emissions tracking of machine learning models as the first step towards sustainable AI
- DASC: Decay-Aware State Compression for Hybrid Linear-Attention Serving
- Layer-Adaptive State Pruning for Deep State Space Models
- 2SSP: A Two-Stage Framework for Structured Pruning of LLMs
- EcoThink: A Green Adaptive Inference Framework for Sustainable and Accessible Agents
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- Carburacy: Summarization Models Tuning and Comparison in Eco-Sustainable Regimes with a Novel Carbon-Aware Accuracy