Searching Where Attention Belongs: Phase-Structured Architecture Search for Small Language Models

Trained language models are built from identical stacked blocks, yet they do not use those blocks identically. Probing studies show that the lower half of a model uses attention to gather information from earlier tokens. The upper half mostly works on what has already been gathered, and its attention can often be skipped with little loss. Architecture search is the automated design of network shapes, and for language models it has largely ignored this. The most efficient language-model search method treats layers as interchangeable, because total parameter count predicts perplexity almost perfectly. Perplexity measures how well the model predicts the next word, and a separate line of evidence says perplexity can rank designs in the opposite order from reasoning ability. So we cannot currently tell whether where capacity sits in the stack matters for what a model can do. We also lack a cheap way to find out.

This note proposes a search space with only a handful of knobs, each tied to a known functional role. One knob is the depth above which attention is removed. Others control how the freed parameters are split between wider middle feed-forward layers and extra attention heads in the lower layers. Candidates are ranked by how cleanly a partially trained model separates tasks in its middle layers, rather than by perplexity or parameter count. The claim is that grounding the search space in how trained models actually behave cuts search cost more than a cleverer scoring rule on an unconstrained space would. The claim also predicts that architectures with matched parameter counts differ on in-context learning and reasoning even when their perplexities match. If that spread does not appear, the parameter-count view wins, and the experiments are designed so that outcome is visible.

Core Claim

At a fixed parameter budget, perplexity is insensitive to layer layout, as LiteTransformerSearch found. That paper reports Spearman correlations of 0.97–0.99 between decoder parameter count and trained perplexity across more than 2,900 models. It then searched a heterogeneous space of over $10^{54}$ configurations by treating parameters as interchangeable.

Downstream capability is a different matter. Rethinking Layer Redundancy in Large Language Models prunes layers from 8B models and finds that perplexity-based and reasoning-based rankings of pruned models correlate at Spearman $\rho = -0.78$. The layers each objective chooses to remove also differ: perplexity calibration removes contiguous mid-to-late blocks, while reasoning calibration removes layers spread across the stack. That result is about removing layers from finished models, not about designing new ones.

Our hypothesis is that the same decoupling holds when models are built from scratch. Specifically, the height of the attention cutoff and the width of the middle layers should move in-context learning and reasoning accuracy while leaving perplexity nearly flat.

The second part of the claim is that a cheap signal can track this. The signal is task decodability, the k-nearest-neighbour separability of task identity in intermediate activations. Emergence and Effectiveness of Task Vectors introduced this measure and showed it peaks in middle layers and tracks in-context learning performance. We propose measuring it on each candidate after a short fraction of its training run.

We deliberately do not use a proxy computed at initialization. Functional phases emerge through training, so depth-weighting initialization statistics in the style of Training-free Transformer Architecture Search has no principled target. That method sums attention-diversity and feed-forward-saliency scores uniformly across layers at initialization.

Research Program

Measuring the phases at small scale. Phase evidence comes almost entirely from 6–8B models. The first step is therefore to check whether the phases exist at our target scale of roughly 350M parameters, which matches the OPT-350M comparison in LiteTransformerSearch. We will train a uniform baseline on The Pile. We will then apply the four stress tests of Attend First, Consolidate Later: noise injection, state freezing, token patching and attention skipping. That work found a sharp gathering-to-consolidation boundary near half depth in Llama2-7B and Mistral-7B. We will also compute the middle-layer similarity profile used in Layer by Layer, which found shared, transition and refinement zones in Llama 2 7B. Finding where the boundary sits at 350M is a useful result on its own, because it fills the open question about scale in the phase literature. The measured boundary also sets the centre of the search ranges.

Search space. Each candidate is described by four numbers:

  • the cutoff depth above which blocks contain only feed-forward layers (no attention and no key-value cache);
  • the fraction of freed parameters moved into middle-zone feed-forward width;
  • the fraction moved into extra lower-layer attention heads;
  • the depth of the widened middle zone.

Every candidate is resized to the same total parameter count, so parameter count cannot tell candidates apart. This space borrows the depth-differentiated philosophy of The Heterogeneity Hypothesis, which showed that layer-differentiated CNN widths found at initialization beat uniform ones. It also borrows from Stage-Wise Neural Architecture Search, which found that middle stages of CNNs deserve disproportionate depth. Here the stages are defined by function rather than by spatial resolution, which Transformers do not have.

Evaluation fidelity. Each candidate is trained for a small fixed fraction of the full token budget. We then score it three ways: perplexity on a held-out set, task decodability at its middle layers, and on-device latency and peak memory profiled on the target hardware as in LiteTransformerSearch. Task decodability is measured on in-context prompts for part-of-speech tagging (Penn Treebank) and bitwise arithmetic, the tasks used by Han et al.

The total search budget is capped at roughly one to one-and-a-half full training runs, which is about two dozen candidates at 5% fidelity. That compute is reported alongside every result, so the cost of search can be weighed directly against any inference savings. A small number of candidates can be enough because the space is low-dimensional. In this setting the search spends compute on a few meaningful directions instead of sampling a vast unconstrained space.

Baselines

The uniform reference is a standard GPT-style model with attention and equal widths in every layer, trained at the same parameter count, data and token budget. It is the "do nothing" point. Any searched architecture must beat it on downstream accuracy at equal or lower latency, or match it at clearly lower latency and key-value memory.

The strongest existing method is LiteTransformerSearch. We will run its evolutionary search over its own per-layer heterogeneous space, using its parameter-count proxy and the same on-device latency profiling and constraint. Its chosen architectures are then trained with our data and tokens. The published number to keep in mind is its perplexity-proxy rank correlation of 0.97–0.99, together with the report that its models outperformed OPT-350M in average accuracy across 14 tasks while running up to 1.6x faster. No reported result was found on Lacuna for a phase-structured or task-decodability-guided language-model search at this scale. Our bar is therefore head-to-head: at matched latency, beat its selected model on mean accuracy over HellaSwag, PIQA, ARC, WinoGrande, BoolQ and LAMBADA, plus the two in-context tasks, by more than seed-to-seed spread measured over three seeds.

The low-fidelity perplexity baseline runs the same short-training evaluation as our method but ranks candidates by early perplexity. It follows the budgeted estimation line in Rethinking Performance Estimation in Neural Architecture Search. It uses exactly our candidates and compute, so any gap isolates the choice of signal.

The random-structure ablation draws the same number of candidates from a four-knob space of equal size. Its knobs reallocate parameters between arbitrary depth ranges instead of phase-aligned ones, and candidates are scored identically. This isolates the key ingredient, which is whether aligning the search space with the phases matters or whether heterogeneity alone is enough.

Finally, the post-hoc skipping baseline takes the fully trained uniform model and removes attention above its measured boundary, following Ben Artzy and Schwartz. This comparison is made at matched inference key-value cache size.

Why not just…?

Why not just use parameter count, as LiteTransformerSearch does? For perplexity it is nearly perfect, and we expect to reproduce that. The pruning results of Kim et al. show that perplexity-driven choices can invert reasoning rankings. We have not seen that tested for architectures trained from scratch. The experiment settles it directly. Train a fixed-parameter slice of the space to completion. If downstream accuracy varies no more than seed noise across layouts, parameter count is sufficient and this proposal's premise fails. We will report that outcome as a negative result.

Why not just train a uniform model and skip upper-layer attention afterward? This is a real threat. Ben Artzy and Schwartz report zero accuracy loss on Capitals and SQuAD after freezing Llama2-7B states past the halfway point. Post-hoc skipping gives the key-value savings without paying for search. It cannot, however, move the freed capacity somewhere useful during training. The post-hoc skipping baseline tests exactly this. If it matches our best candidate on accuracy at equal cache size, the reallocation idea adds nothing beyond the phase measurement itself.

Why not just use early perplexity or an off-the-shelf zero-cost proxy? Off-the-shelf proxies such as SNIP and NASWOT already fail on Vision Transformers, according to TF-TAS and Auto-Prox. In Auto-Prox, an evolved proxy reached a Kendall's tau of 82.07% versus 63.83% for TF-TAS on CIFAR-100. Early perplexity is the more serious rival. The low-fidelity perplexity baseline compares against it at identical cost.

Why not just learn widths during training? Adaptive Width Neural Networks learns layer widths within a single training run. It has not been extended to attention heads or to deciding whether an attention sublayer should exist. It also requires bounded activations, which conflicts with SwiGLU-style feed-forward blocks. We therefore treat it as future work rather than as a baseline.

Decisive Checks

Three results decide the proposal.

The first is spread. Among fully trained candidates with matched parameter counts, downstream accuracy must vary by more than seed noise while perplexity stays nearly constant. Without this spread, the proposal stops.

The second is ranking. On a held-out set of candidates trained to completion, early task decodability must rank final downstream accuracy better than both early perplexity and parameter count, measured by Spearman correlation.

The third is the final comparison. The best phase-structured model must beat both LiteTransformerSearch's selection and post-hoc attention skipping, at matched latency and key-value memory. That advantage must hold with search compute included in the accounting.