Deployers usually want a smaller, faster language model that is good at one thing, such as math or code, rather than a model that is merely good at predicting text. Architecture search is the automated hunt for the best network shape under a speed or memory budget. For language models it has mostly relied on cheap scores that predict overall text-prediction quality. The best-known cheap score is simply the parameter count. Interpretability research points the other way: specific skills live in tiny "circuits," meaning small sets of attention heads and neurons. Deleting five of the right heads can wipe out most of a model's math ability while the parameter count barely changes. This note proposes to map those skill circuits once, on the full model, and then score thousands of candidate compressed models against that map with no forward passes at all. Each candidate's score is a lookup of how much of the circuit it keeps, corrected for how circuit parts interact.
The claim that makes this worth testing is narrow and falsifiable. For task-specific quality, a candidate's retention of a circuit located once on the parent model predicts how that candidate ranks against other candidates. It does so better than parameter count and better than importance scores computed on task data alone, and it keeps that advantage among the top candidates, where existing cheap scores are known to fail. Because the expensive localization step is paid once and spread across the whole search, the search itself costs essentially nothing per candidate.
What the two directions supply, and the gap between them
The architecture-search side has made search cheap by avoiding training. LiteTransformerSearch trained over 2,900 decoder models and found that parameter count ranks their converged perplexity with a Spearman correlation of 0.97 to 0.99. Layer shape affected speed but not quality, so it could trace speed–memory–quality tradeoff curves on a laptop CPU in under three hours. LPZero evolves symbolic scoring formulas over activations, gradients, and attention tensors, and uses one to guide pruning of LLaMA-7B. EZNAS does the same for vision networks and reports that these cheap scores stop discriminating among the top 10% of architectures. Compressing Large Language Models with Automated Sub-Network Search treats a pretrained model as a pool of sub-networks over layers, heads, and MLP widths. It orders units by activation magnitude and trains the pool with distillation, at roughly four times the cost of ordinary training.
All of these methods either score global statistics or target perplexity, and our proposal replaces both with a task-specific causal map. Parameter count in particular was only ever validated against perplexity.
The interpretability side shows that task skills are sparse and can be found cheaply. Compressed sensing for capability localization ablates random groups of heads on about 100 examples and fits a sparse regression. It finds task-critical heads with up to 50× fewer evaluations than one-at-a-time ablation. Removing five such heads cuts GSM8K by up to 60% while unrelated tasks drop under 2%. The same paper also shows that task heads are necessary but not sufficient: a set of "universal" heads must also survive. MathNeuro uses only forward passes to find the weights that matter more on math prompts than on general text, which amount to 1.5–1.8% of weights. Zeroing them sends math accuracy to near zero while MMLU and RACE are largely untouched. Contextual decomposition (CD-T) recovers hand-built circuits with 97% ROC AUC in seconds.
These papers motivate the proposal and supply its localization methods. They also constrain it. Unstructured weight sets like MathNeuro's give no speedup on ordinary hardware. Edge pruning finds very sparse circuits but needed 32 H100 GPUs for a 13B model, so it is too costly to sit inside a search loop. P2D is the closest bridge. It locates task heads quickly and fine-tunes only those, but it keeps the architecture fixed and freezes every MLP. None of these papers uses a circuit map to rank structured, hardware-executable candidates.
Core Claim
Let a candidate sub-network be a binary mask $m$ over structured units: attention heads, MLP intermediate channels, and whole layers. For a target task $t$ we fit a score on the parent model only:
$$S_t(m) = \sum_i w^t_i\, m_i + \sum_{(i,j)\in P} v^t_{ij}\, m_i m_j .$$
Here $w^t_i$ is the estimated effect of unit $i$ on the task metric, and $v^t_{ij}$ captures pairwise interactions for a small set $P$ of candidate pairs. An example of such an interaction is a backup head that compensates when its primary head is removed. Both are fitted from random structured ablations of the parent, extending the compressed-sensing regression from heads to MLP channel groups and adding interaction terms.
We fit the same form a second time on general-text data to get $S_g(m)$, which represents the universal units. The search then maximizes $S_t(m)$ subject to a measured latency budget and a floor $S_g(m) \ge \tau$, where $\tau$ is a chosen fraction of the parent's general score. Evaluating $S_t$ for a new mask is a dot product. All model forward passes happen during the one-time fit.
The prediction is that $S_t$ ranks candidates by their actual task accuracy with high Kendall's tau, both before and after a short fixed recovery fine-tune. The prediction also says the interaction terms matter most in the top decile.
Research Program
The work has three parts, each of which can fail separately.
The first part tests whether parameter count breaks for task quality. We sample several thousand structured sub-networks of the parent at matched parameter counts but different circuit coverage. On these we measure GSM8K, MBPP, MMLU, and BoolQ accuracy, all without recovery training. If accuracy at fixed size varies little, the tension between size-only search and circuit localization disappears for structured compression, and we stop.
The second part builds the map. Head effects come from compressed-sensing ablation. MLP effects come from MathNeuro-style task-minus-general importance, aggregated from individual weights to intermediate channels. Aggregation may dilute a signal that is spread thinly across many weights, so we check it directly: channel-level zeroing should reproduce MathNeuro's selective math collapse. We fit interaction terms only for pairs whose joint ablation deviates from the additive prediction, which keeps $P$ small. CD-T serves as an independent localizer for cross-checking the heads.
The third part runs the search. We use evolutionary search in the style of LiteTransformerSearch over the joint space of layers, heads, and MLP width used by the sub-network search paper, with latency measured on the target GPU.
For validation we use GPT-2 Small on indirect object identification, where the 28-head IOI circuit and its backup and negative heads are known. This lets us check whether the fitted interaction terms recover the known backup behaviour. The main experiments use LLaMA-7B, for comparability with LPZero's pruning result, with math and code as the target tasks.
Baselines
The naive reference points are the unpruned parent and random structured pruning at matched latency. A method that cannot beat random pruning at the same speed has learned nothing about where the skill lives.
The size-only reference is parameter count as used by LiteTransformerSearch. It runs inside the identical evolutionary search with the identical latency model, with only the scoring function swapped. Its reported strength is a 0.97–0.99 Spearman correlation with perplexity. Beating it means a higher task-accuracy Kendall's tau and a better accuracy-versus-latency curve on GSM8K and MBPP.
The strongest existing competitor is activation-magnitude ordering from Automated Sub-Network Search. We run it with task-mixture calibration data so that it is task-aware, not generic. The research file reports a +9.85% downstream accuracy improvement and up to 22% latency speedup for it, but no per-benchmark GSM8K figure appears there. To keep the comparison fair, we evaluate it two ways. The first uses its full distillation recipe at its roughly 4× training cost. The second uses only its ordering, with the same short LoRA recovery we apply to our candidates. Beating it means matching or exceeding its task accuracy at equal latency for less total GPU time, counting our localization cost.
The global cheap-score competitor is an LPZero proxy. It reports an average of 64.18 across 11 commonsense tasks for LLaMA-7B pruning, and that number is only comparable on those commonsense tasks, not on math or code. We use its published formula and, as a separate variant, re-evolve it for our tasks, charging our method the roughly 10 GPU hours LPZero's search needs.
Three ablations isolate the key ingredients. The first removes the general-text term, leaving task importance only. The second drops the interaction terms, which gives the purely additive compressed-sensing assumption. The third drops the universal-unit floor. All methods get the same calibration examples: 100 per task, matching the compressed-sensing setup.
Why not just…?
Why not just use task-calibrated activation or Wanda-style magnitude scores? This is the most serious threat. Magnitude on task data already approximates "units that are busy on math." It does not separate units that are busy on math from units that are busy on everything. MathNeuro's selectivity depends on exactly that subtraction of general-text importance, and P2D reports that generic parameter selection underperforms task-matched selection. Magnitude also cannot see backup or negative heads, which matter causally without being loud. The first ablation, which removes the general-text term, is exactly this baseline. If it matches the full method, the honest conclusion is that the causal machinery adds nothing, and we will report that.
Why not just evaluate every candidate on 100 task examples? This is low-fidelity direct evaluation. It is plausibly as accurate as our score, and possibly more accurate, because it measures the candidate itself. Our method's advantage is cost. Direct evaluation costs a full batch of generation per candidate, often multi-step reasoning, and the search visits thousands of candidates. Our map is fitted once. We will report accuracy of the chosen model against total GPU-hours for both methods. If direct evaluation reaches the same final model at comparable cost, the contribution shrinks to cases where search budgets are large. A hybrid, in which our score prunes candidates and direct evaluation reranks the top 5%, is the likely practical winner and is included.
Why not just count parameters? The file's evidence that a few heads can remove most of GSM8K accuracy suggests size cannot rank task skill among equal-size structured candidates. But that evidence comes from targeted ablations, not from the random-shape candidates a search actually visits. The fixed-size experiment in the first part of the program settles this directly.
Why not just use an evolved proxy like LPZero? It scores global statistics of each candidate and needs forward and backward passes per candidate, plus a search of about 10 GPU hours to evolve the formula. It has no notion of which units carry a task. The top-decile comparison is where we expect the difference to show, since global scores are known to blur the best candidates.
Proposed Experiments and Decisive Checks
For each target task, the main measurement is the Kendall's tau between each score and the true accuracy of 2,000 sampled candidates. We report it over all candidates and within the top decile, measured without recovery training and after a fixed 200-step LoRA recovery. Off-target preservation is measured on MMLU and BoolQ. Latency and GPU-hours are reported end to end, including localization.
The GPT-2 IOI check passes if interaction terms fitted blind recover the known backup heads. The decisive LLaMA-7B result is a top-decile tau advantage over task-calibrated magnitude ordering, together with a latency-matched GSM8K and MBPP gain over it at lower total compute. If recovery fine-tuning erases the ranking advantage, circuit maps matter for zero-shot compression but not for compression with recovery training, and that is a useful negative result.
References
- LiteTransformerSearch: Training-free Neural Architecture Search for Efficient Language Models
- LPZero: Language Model Zero-cost Proxy Search from Zero
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Compressing Large Language Models with Automated Sub-Network Search
- Compressed Sensing for Capability Localization in Large Language Models
- Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes
- Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition
- Finding Transformer Circuits With Edge Pruning
- From Parameters to Data: A Task-Parameter-Guided Fine-Tuning Pipeline for Efficient LLM Alignment
- Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 Small