Large language models spend most of their parameters in feed-forward layers. These are wide two-layer networks inside each block that expand every token's representation into thousands of "intermediate neurons" and then project it back down. Removing whole intermediate neurons is the kind of pruning that actually speeds up ordinary matrix-multiply hardware. It also does real damage, and the damage falls hardest on stored factual knowledge. This note proposes a different trade. We do not throw a pruned neuron away. We regenerate its activation almost for free from a handful of neurons we kept, looking at the current token and the few tokens just before it. We also turn this around and use it to choose what to prune. A neuron whose activation can be predicted cheaply from its neighbors is a good candidate for removal. A neuron that cannot be predicted carries information nothing else has, and it should stay.
The idea borrows from Ghost modules in vision. A Ghost module computes a small set of "intrinsic" feature maps with expensive operations and derives the remaining "ghost" maps through cheap ones, usually depthwise convolutions, which filter each channel separately. In two sentences, for a colleague: structurally prune a decoder LLM's feed-forward neurons, but rebuild each pruned neuron's pre-activation with a tiny causal filter over a few surviving neurons, at roughly a few hundredths of a percent of the original cost per neuron. Pick which neurons to prune by how well they can be rebuilt this way, not by weight magnitude.
Core Claim
We claim that a large fraction of feed-forward pre-activations in decoder LLMs can be predicted by a sparse, short-memory, causal function of other pre-activations. Regenerating them before the layer's nonlinearity recovers a meaningful share of the accuracy that structured pruning loses, at matched parameter removal and with real wall-clock speedups. Let $g_{j,t}$ and $u_{j,t}$ be the gate and up pre-activations of kept neuron $j$ at token $t$. Each pruned neuron $p$ gets a small support set $S_p$ of $s$ kept neurons and a causal kernel of length $k$, and is regenerated as
$$\tilde g_{p,t} = \sum_{j \in S_p} \sum_{\tau=0}^{k-1} w_{p,j,\tau}\, g_{j,t-\tau}, \qquad \tilde h_{p,t} = \mathrm{SiLU}(\tilde g_{p,t})\, \tilde u_{p,t},$$
where $\tilde u_{p,t}$ is built the same way from the $u_j$ with its own weights. $\tilde h_{p,t}$ then enters the original down-projection row of neuron $p$. With $s=4$ and $k=4$, a ghost neuron costs about 32 multiply-adds per token for each of gate and up. The neuron it replaces costs $d_{model}$ (4096 in Llama-2-7B) for each, so the ghost is roughly 128 times cheaper. During decoding, the only extra state is a rolling buffer of the last $k-1$ pre-activations of the support neurons.
Two design choices separate this from things that already exist. First, we regenerate before the nonlinearity. A linear map from kept post-activation outputs to pruned outputs would fold exactly into a refitted down-projection, so it would amount to ordinary "prune and re-solve least squares." Second, the temporal kernel uses preceding tokens, which no channel-only compensation can do. If neither ingredient helps, the method reduces to known baselines, and the experiments are built to show whether that happens.
Relation to Prior Work
The proposal reuses the Ghost mechanism from GhostNet: More Features From Cheap Operations. It is motivated most directly by GhostBERT: Generate More Features with Cheap Operations for BERT. GhostBERT generated ghost attention and feed-forward features in BERT-style encoders using one-dimensional depthwise convolutions with softmax-normalized kernels. On GLUE it retained 99.6% of BERT-base performance at 50% of the FLOPs.
GhostBERT differs from this proposal in three ways. Its convolutions are bidirectional, so they are invalid for next-token decoding. It relies on full distillation and fine-tuning rather than a post-training budget. It uses a uniform width multiplier rather than deciding which neurons become ghosts. We change all three: the kernels are causal, the ghost weights are fit post-training, and the intrinsic/ghost split is chosen per neuron by predictability.
Two papers motivate the gap. LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment reports that sparsification damages knowledge-intensive recall far more than quantization does, and that structured sparsity patterns often lack kernel support. Restoring Pruned Large Language Models via Lost Component Compensation (RestoreLCC) compensates pruned attention heads with a static vector absorbed into the bias. It explicitly leaves feed-forward compensation open, and a static vector cannot depend on the input. Our ghost neurons depend on the input and target exactly that feed-forward gap.
Three papers constrain the design, and two of them serve as baselines. EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation adds a low-rank branch projected into the activation eigenspace. It needs a fused kernel to stay fast and mixes only channels, never tokens. ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization wraps a 2:4 sparse core in block-diagonal matrices. Unfused, it reached only 1.57× speedup against a theoretical 2.0×. That result warns us that cheap-looking extra operations can eat the speedup, so we profile latency from the start.
At the coarse end, Streamlining Redundant Layers to Compress Large Language Models (LLM-Streamline) removes contiguous blocks of layers and trains a small replacement network by hidden-state MSE distillation. What Matters in Transformers? Not All Attention is Needed drops whole attention layers. It also shows that feed-forward layers are much less redundant than attention, which is a warning for us. If feed-forward neurons were simply redundant, dropping them would be easy. Our bet is narrower: the redundancy that matters is redundancy conditional on cheap computation from neighbors, and that is the quantity we measure.
Research Program
The work begins with a measurement study that can kill the project early. On Llama-2-7B and Llama-3.1-8B, using C4 and SlimPajama calibration text, we compute, for every feed-forward neuron in every layer, how much of the variance in its gate and up pre-activations is explained by three predictors of increasing power. The first uses $s$ kept neurons at the same token. The second adds a causal window of $k$ tokens. The third is a dense least-squares fit on all kept neurons, which serves as an upper reference.
The supports $S_p$ are chosen greedily by correlation. The study produces per-layer distributions of predictability and tests two things. It checks whether the temporal window adds explained variance beyond same-token prediction. It also checks whether predictability varies across depth enough to justify allocating the ghost ratio per layer rather than uniformly, the open problem the research file flags for Ghost architectures.
Next we build the pruning rule. Each neuron gets a score: its unexplained variance multiplied by the squared norm of its down-projection row, which estimates the output error its ghost would leave behind. We prune the lowest-scoring neurons, allocating across layers greedily to minimize total estimated error at a target parameter budget. This rule should differ systematically from magnitude- or activation-based rules. In particular, we expect rarely firing, idiosyncratic neurons, plausibly the ones storing long-tail facts, to score as unpredictable and therefore be kept. That is how we hope to blunt the knowledge loss LLMCBench attributes to pruning.
The ghost weights are then fit layer by layer, first in closed form by least squares on pre-activations. Optionally they are refined with a few hours of hidden-state MSE distillation on 30,000 SlimPajama samples, the same budget LLM-Streamline uses. The down-projection of the kept neurons may also be refit jointly. Finally, we write a fused Triton kernel that gathers support pre-activations, applies the causal filter, gates, and feeds the down-projection. We report prefill and decode throughput against the plain pruned model. Decoding is bandwidth-bound and the ghost weights are tiny, so we expect the ghost overhead to be small. That expectation is a hypothesis to measure, not an assumption.
Baselines
The naive reference is the dense model, along with the same structurally pruned model with the pruned neurons simply deleted and no compensation, using the identical neuron selection. That gap is the deficit everything else tries to close.
The most important ablation, and the competitor most likely to absorb the idea, is prune-and-refit. Here the pruned neurons' post-activation outputs are reconstructed by least squares from all kept neurons, and that map is folded into the down-projection. It costs nothing at inference and needs no new kernel. Ghost neurons count as a contribution only if they beat this at equal parameter count. Two narrower ablations isolate the ingredients. A same-token ghost ($k=1$) tests whether the temporal window matters. A magnitude-selected pruning mask with the same ghost generator tests whether "prune what you can regenerate" matters as a selection rule.
The strongest published coarse-grained method is LLM-Streamline. It reports 93.1% relative accuracy on Llama-2-7B with about 25% of parameters removed, against roughly 78% for LLM-Pruner and SliceGPT (Streamlining Redundant Layers to Compress Large Language Models). Feed-forward layers hold about two-thirds of Llama-2-7B's parameters, so removing about 37% of intermediate neurons reaches the same 25% budget. We rerun LLM-Streamline ourselves with its 30,000-sample SlimPajama protocol, on the same benchmark suite and with the same single-GPU training budget (about 5 hours on one A800-class GPU, as reported). Beating it means higher relative accuracy at matched removed parameters and matched measured latency, not just matched FLOPs.
For input-dependent compensation we compare against EoRA. Its low-rank branch is applied to the feed-forward layers of the pruned model, with the rank chosen so that its added parameters equal those of the ghost generators. EoRA's own reported gains are for 3-bit and 2:4 compression of LLaMA3-8B: 11.45% on GSM8K and 10.84% on ARC-Challenge at 3 bits, and 11.83% on GSM8K at 2:4. The file has no reported result on Lacuna for EoRA under structured neuron pruning, so we generate that number ourselves.
RestoreLCC's static-bias idea, extended to feed-forward layers, serves as the zero-overhead reference. Its reported result is a +2.01% mean accuracy gain over the best baseline on LLaMA-7B at 50% unstructured sparsity. LoRA fine-tuning of the pruned model under the same data and GPU-hour budget is the generic-recovery reference.
Why not just…?
Why not just prune neurons and refit the down-projection by least squares? This is the real threat. If pruned neurons are predictable at all, a linear refit already captures their linear predictable part for free. The argument for ghosts depends on two things the refit cannot do: predict through the nonlinearity, and use preceding tokens. The measurement study answers this question before any kernel work. If the window-$k$, pre-activation predictor explains little more variance than the folded linear one, and the end-to-end comparison confirms it, we drop the ghost generator and report "predictability-based selection plus refit" as the result.
Why not just drop whole layers or attention modules instead? The coarse methods are fast and simple. But dropping 50% of attention layers on Llama-2-70B still costs 2.4% average accuracy (What Matters in Transformers? Not All Attention is Needed). Bridged block removal reaches 93.1% relative accuracy, not parity. Neither method reaches the feed-forward layers, where most of the parameters live. Our method is orthogonal to attention dropping, so we also test the two stacked. The head-to-head comparison against LLM-Streamline at the 25% budget settles which granularity loses less.
Why not just quantize to 4 bits or below and compensate with EoRA? Quantization keeps knowledge better than pruning, according to LLMCBench, and it may well beat us at equal memory. We do not claim otherwise. The claim is that structured removal and quantization stack. We therefore compare ghost-pruned-plus-4-bit models against 3-bit-plus-EoRA models at equal weight memory, on knowledge-heavy MMLU and on GSM8K separately. If 3-bit-plus-EoRA dominates on both, the stacking argument fails at that memory point and we report it.
Why not just LoRA-fine-tune the pruned model? RestoreLCC reports that PEFT methods often fail to converge at 60% sparsity. LoRA also adds a dense side branch at inference unless it is merged. Merged, it is a weight update with no ability to use token context. We give LoRA the same data and GPU hours and report both merged and unmerged latency.
Decisive Experiments
The first decisive check is the predictability study. We need the causal window to add measurable explained variance over same-token prediction, and we need predictability-weighted selection to keep a different set of neurons than magnitude selection. If both fail, the project narrows to selection plus refit.
The main experiment runs on Llama-2-7B, Llama-3.1-8B, and one Qwen model at 25% and 40% parameter removal. It reports BoolQ, HellaSwag, PIQA, WinoGrande, ARC-Challenge, MMLU, and GSM8K accuracy, WikiText-2 and C4 perplexity, and LLM-Streamline's Stability metric, which catches models that keep accuracy by erratic guessing. All baselines above share calibration data and compute.
The split between MMLU and the reasoning tasks tests a specific prediction. If unpredictable neurons hold knowledge, predictability-based selection should close much more of the MMLU gap than magnitude selection, while the reasoning benchmarks change less.
Finally, decode and prefill throughput on A100-class hardware, at batch sizes 1 and 32, must show that the ghost model keeps most of the plain pruned model's speedup. We treat keeping less than ARMOR's roughly 84% of theoretical speedup as a failure of the systems side.
References
- GhostNet: More Features From Cheap Operations
- GhostBERT: Generate More Features with Cheap Operations for BERT
- LLMCBench: Benchmarking Large Language Model Compression for Efficient Deployment
- Restoring Pruned Large Language Models via Lost Component Compensation
- EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation
- ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
- Streamlining Redundant Layers to Compress Large Language Models
- What Matters in Transformers? Not All Attention is Needed