Language models are increasingly deployed inside test-time search. Instead of answering once, the model drafts many candidate solutions, and a vote or a learned scorer picks among them. Architecture choices are still made as though the model will answer once. Current methods for choosing a model's shape (width, depth, the split between attention and feed-forward layers, how many attention heads share memory) rank candidates by pretraining loss and single-pass generation speed.
This proposal asks whether that ranking survives when the model is used as a search engine. Search changes two things. It changes what one generated candidate costs, because many branches share a prompt and each keeps its own memory state. It may also change how fast accuracy grows as more candidates are drawn, which is a property of the model's output distribution that loss may not capture. If two architectures with equal loss improve at different rates under repeated sampling, then loss-based architecture selection picks the wrong model for search-heavy deployment. A cheap way to measure that rate early in training would then be worth building.
Core Claim
The claim has two parts, and either can fail. First, at matched pretraining compute and matched validation loss, architectural shape measurably changes the exponent of the sampling curve. That curve is how quickly the fraction of unsolved problems falls as the number of sampled candidates $k$ grows. Second, this exponent can be estimated from short, budgeted training runs, cheaply enough that folding it into architecture selection moves the cost-accuracy frontier of search-based deployment.
Following the power-law decay of errors with sample count reported by Kwok et al., we model the unsolved fraction of a benchmark under an architecture $a$ as
$$ E_a(k) \approx c_a \, k^{-\beta_a}, $$
where $k$ is the number of sampled candidates, $c_a$ is the single-sample error, and $\beta_a$ is the rate at which extra samples help. Loss-based architecture laws effectively predict $c_a$. The question is whether $\beta_a$ carries independent architectural information.
The lifetime objective extends the total-cost view of Beyond Chinchilla-Optimal:
$$ C_{\text{total}} = 6ND + Q \, k \, \kappa_a , $$
where $N$ is parameter count, $D$ is training tokens, $Q$ is the number of deployed queries, and $\kappa_a$ is the measured cost of one search candidate under architecture $a$. That per-candidate cost includes shared-prompt reuse, per-branch memory for past attention states, and scorer calls. That paper prices inference as one autoregressive pass per query. Here the inference term depends on the search policy and on the architecture.
Where This Sits
The motivating evidence comes from three papers.
Inference Scaling Laws shows that small models paired with reward-guided tree search (REBASE) can beat much larger models that only sample. It treats architectures as fixed off-the-shelf checkpoints, so it cannot say whether a different shape at the same size would search better.
Scaling Laws Meet Model Architecture supplies the method we build on. It fits a loss law calibrated by hidden size, the feed-forward-to-attention ratio, and grouped-query attention (several query heads sharing one stored key/value set). It then searches for architectures on a single-pass throughput frontier. It never measures behaviour under repeated sampling, and it evaluates on zero-shot benchmarks rather than multi-step reasoning.
Dynamic Experts Search is the most direct hint that architecture shapes coverage. Varying how many experts are active in a mixture-of-experts model yields correct-answer sets with Jaccard overlap of only about 0.5–0.6 on GSM8K, and lower on MATH500, while average accuracy stays flat. That paper exploits the effect at runtime inside a fixed trained model. We ask whether the same effect should shape design before training.
Two other lines of work constrain the design.
The first is architecture-search cost. Zero-cost proxies such as EZNAS score untrained networks and reach Kendall Tau 0.65 on NASBench-201, but struggle to separate the top 10% of candidates. That is exactly the regime where frontier architectures differ. Rethinking Performance Estimation instead finds the smallest training budget that preserves rank correlation with full training. We reuse that budgeted-estimation idea, but the target is the sampling exponent rather than final accuracy.
The second is search that goes wrong. More Test-Time Compute Can Hurt and Inverse Scaling in Test-Time Compute show that scorer-guided search can get worse with more compute. We therefore measure the exponent under verifier-free majority voting as well as under a learned scorer, so a flaw in the scorer is not mistaken for an architecture effect.
Pipeline-level search such as ARCHON composes black-box model calls and reports that 7B models fail in its critique and fusion roles. It never changes a network's internals, so it is complementary to this work rather than a competitor.
Research Program
The work has three stages.
Stage 1: a controlled architecture grid. We train roughly fifty decoder-only models on FineWeb mixed with a fixed fraction of math-heavy text.
- Compute levels: three pretraining budgets, spanning roughly 100M to 1B parameters.
- Varied factors: depth-to-width ratio, feed-forward-to-attention parameter ratio, and grouped-query sharing factor.
- Fixed factors: tokenizer, data order, and optimizer recipe.
- Seeds: two seeds for a subset, to measure noise in the fitted exponent.
Every model then receives the same short supervised fine-tune on step-by-step math solutions. For each model we fit $c_a$ and $\beta_a$ on GSM8K and MATH500 with $k$ from 1 to 256, under three selection rules:
- oracle coverage (pass@k),
- majority voting,
- a fixed small outcome scorer trained once, in the style of Training Verifiers to Solve Math Word Problems, and shared across all generators.
In parallel, we measure the per-candidate cost $\kappa_a$ on real hardware. It is recorded in FLOPs and in wall-clock time at fixed memory, with shared-prefix caching and batched branches enabled. This captures an interaction that Scaling Laws Meet Model Architecture could not see: grouped-query sharing shrinks per-branch memory. That lets more branches fit on a device, so its payoff under search may differ from its payoff in single-pass decoding.
Stage 2: a search-aware architecture law. We augment the existing loss calibration with a term predicting $\beta_a$ from the same architectural variables. We then ask whether the joint law reorders the frontier of accuracy against $C_{\text{total}}$ at fixed query volume $Q$.
Stage 3: a cheap estimator. This is the architecture-search-cost stage. We evaluate pass@k at $k \le 32$ on a 200-problem held-out set from early checkpoints (5%, 10%, 25% of training). We report the smallest training fraction at which the estimated exponents keep their Spearman rank against final exponents. This mirrors how Rethinking Performance Estimation treats accuracy. The compute spent on this estimation is charged against any gains we claim.
Baselines
Default dense shape. The naive reference is a standard dense model at the conventional Llama-style depth-to-width ratio. It is trained at the same total pretraining compute, including an overtrained variant as recommended by Beyond Chinchilla-Optimal for high query volume. It is then paired with the same search policies. This is the "do nothing to the architecture" point. Any proposed architecture must beat it at equal $C_{\text{total}}$, and the estimation overhead counts against the proposal.
Loss-and-throughput selection. The strongest competitor is architecture selection by the loss-plus-throughput law of Scaling Laws Meet Model Architecture. That paper reports Panda-1B at 2.1% higher accuracy than LLaMA-3.2-1B on zero-shot benchmarks under identical training compute. It has no reported search-based reasoning result on Lacuna, and we found none for GSM8K or MATH500 at our model scales.
- Fairness: we refit this law on our own grid so it sees identical data, then let it choose its frontier architecture at each budget.
- Win condition: the search-aware law must pick a different architecture, and that architecture must deliver higher accuracy at equal total cost. A gain of a few points of GSM8K accuracy at matched cost, outside seed variance, counts as beating it.
Best search, held fixed. To keep the architecture effect separate from search quality, every architecture is also run with the best search methods in the file.
- REBASE, which Inference Scaling Laws reports lets Llemma-7B beat Llemma-34B on MATH500 at 7× less compute.
- DORA, which Every Rollout Counts reports beats REBASE-at-256-rollouts using 64 rollouts, a 3.5× FLOP reduction.
The architecture effect should persist on top of these, not only under naive sampling.
Ablation. The key ablation removes the exponent term from the selection law, keeping only the per-candidate cost model $\kappa_a$. This separates two possible sources of gain: search-aware cost accounting alone, or the claim that architecture changes sampling behaviour.
Why not just…?
Why not just predict search accuracy from loss? This is the real threat. If $\beta_a$ is a function of validation loss alone, the existing law plus a cost model is sufficient, and our exponent term adds nothing. The matched-loss test in Stage 1 settles it directly. If, among architecture pairs within 0.01 nats of each other, the spread in fitted exponents is no larger than the seed-to-seed spread, we report a negative result. The contribution then shrinks to the cost-accounting ablation. The Dynamic Experts Search overlap numbers suggest the spread will be real, but they come from runtime routing in one trained model, not from separately trained architectures.
Why not just overtrain a standard shape and spend the difference on search? That is our first baseline, and it may well win at small query volume $Q$. The claim is only that the gap opens as $Q$ grows. There, per-candidate cost and sampling rate both compound over $k$ candidates and many queries, whereas loss advantages are paid for once.
Why not just tune the search algorithm or route queries per instance? Methods like Latency and Token-Aware Test-Time Compute route each query to the cheapest adequate search strategy, and DORA removes redundant branches. Both act on a fixed generator. We run them on every architecture and report how much of the total gain comes from architecture versus search policy. If architecture differences vanish under DORA, that is an informative finding we will state plainly.
Why not just use zero-cost proxies? Their documented weakness is ranking near the top of the candidate pool (per EZNAS). They also measure single forward and backward passes, which carry no information about the diversity of sampled solutions. We include Synflow-style and EZNAS-style scores as competing estimators of $\beta_a$ in Stage 3 and report their rank correlation against our early-checkpoint estimator.
Decisive Checks
Three results decide the project.
The first is the matched-loss exponent spread described above. Without it, the second half of the core claim is false.
The second is a frontier reordering at realistic query volume. The search-aware law must select a different depth, width, or head-sharing configuration from the loss-based law, and that selection must win at equal $C_{\text{total}}$ under majority voting, the learned scorer, and REBASE. It should hold on GSM8K and, at the largest scale, MATH500.
The third is cost. The exponent must keep its rank (Spearman ≥ 0.8 against the final exponent) at or below 10% of training compute. Otherwise the architecture-search overhead erases the deployment savings, and the practical case collapses even if the scientific effect is real.
References
- RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
- Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs
- Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Rethinking Performance Estimation in Neural Architecture Search
- More Test-Time Compute Can Hurt: Overestimation Bias in LLM Beam Search
- Inverse Scaling in Test-Time Compute
- An Architecture Search Framework for Inference-Time Techniques
- Training Verifiers to Solve Math Word Problems
- Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling
- Latency and Token-Aware Test-Time Compute