Two-Knob Reasoning Probes: Scoring Step Precision and Chain Depth Separately

Reasoning benchmarks for language models usually report one number per problem: was the final answer right? That number mixes two different demands. The first is precision at a single step, such as keeping twelve digits aligned during one multiplication. The second is depth, meaning how many dependent steps the model can chain before something breaks. Sensing physics has a name for this kind of tension, the resolution–penetration trade-off: high-frequency waves resolve fine detail but die out quickly in a medium, and low-frequency waves travel deep but blur. How Data Shapes RoPE Frequency Usage showed that the same tension exists inside transformers. RoPE (rotary position embeddings, which encode token position as rotations at many frequencies) uses high frequencies for fine local position and low frequencies for long range. When those frequencies are stretched to extend context, arithmetic exact-match accuracy falls to near zero even though perplexity improves.

We propose a generated benchmark whose items set step precision and chain depth independently, and which checks every intermediate state against an executable reference rather than only the final answer. The claim is not simply "harder problems." We claim that final-answer benchmarks collapse two capability axes into one difficulty scale. We expect this collapse to explain part of why such benchmarks saturate quickly and why their score matrices look low-rank. Scoring the two axes separately, at the level of individual steps, should recover capability differences that current evaluations cannot see, and it should keep separating new models for longer.

Core Claim

Two observations motivate this proposal. Certifiable Evaluation factors a 157-model by 86-task score matrix and finds an effective rank of about 8, with static suites 80–95% redundant. Evaluating Intermediate Reasoning of Code-Assisted LLMs for Mathematics shows that on MATH500, sound programs and brute-force or ungrounded programs reach the same execution pass rates. If identical final scores can hide different reasoning, then part of the low rank may be an artifact of coarse measurement.

Our hypothesis has three parts, each testable:

  1. Per-step error rates, indexed by step position and by the local precision demand, form a model "attenuation profile."
  2. These profiles have higher effective rank than final-answer accuracy on the same items.
  3. Items defined by where they sit on this profile keep their power to discriminate among later-released models longer than items drawn from static pools.

The physics analogy constrains the design without being taken literally. The RoPE result predicts a specific signature: models whose positional frequencies were downscaled should lose ground on the precision axis much more than on the depth axis. That gives us an external check that the two axes are real.

Research Program

The generator produces items from three task families. In each family, every intermediate state is computable by a short program.

The first family is chained multi-digit arithmetic, the setting where both When Splitting Makes Stronger and the RoPE paper report failures. The second is iGSM-style word problems built from dependency graphs, extending the synthetic iGSM usage in the RoPE paper. The third is discrete state-tracking games of the kind MTR-Bench runs over multiple turns (MTR-Bench).

Each item has two knobs:

  • Width $W$ is the number of tokens that must be aligned exactly within one step, for example operand digit length or the number of fields in a state update. This borrows the dependency-width notion from the RoPE paper's frequency-matching law $\theta^* = \pi/W$.
  • Depth $D$ is the number of sequentially dependent steps.

Models must emit intermediate states in a fixed, parseable format. A checker compares each state to the reference. Verification is programmatic, which avoids the judge problem documented by FaithBench.

The central measurement is a per-model step hazard. It is the probability that step $t$ is the first wrong step, given all earlier steps were correct, on an item of width $W$. We fit it as

$$ \operatorname{logit} h_m(t, W) = a_m + b_m \log W + c_m \log t + d_m \log W \log t, $$

where $m$ indexes the model and the coefficients are fitted per model. In this fit:

  • $b_m$ measures resolution loss, meaning how much wider steps hurt.
  • $c_m$ measures depth attenuation, meaning how error risk grows along the chain.
  • $d_m$ measures whether wide steps attenuate faster.

The log form is a starting assumption, and we will check it against nonparametric binned hazards. Stacking the fitted profiles across models gives a model-by-(cell, step) matrix. On this matrix we repeat the held-out effective-rank and D-optimal analysis of Certifiable Evaluation. We also fit a two-dimensional item response model: a psychometric model in which each item has a difficulty and a discrimination, and each model has an ability. For this we adapt Fluid Language Model Benchmarking from one latent ability to separate precision and depth abilities. Adaptive testing then selects the next $(W, D)$ cell for each model.

The grid also yields a consistency measure for free. A cell with smaller $W$ or $D$ is a strict sub-problem of a larger one, so we can compute the easy-given-hard consistency score of ConsisEval separately along each axis. That tells us whether hard-to-easy failures come mostly from precision slips or from depth slips.

Baselines

Naive reference. Final-answer accuracy averaged over randomly sampled grid items, the way most leaderboards score. It shows how much ranking and rank information one number already carries.

Strongest competitor for obsolescence resistance: Fluid Benchmarking (Fluid Language Model Benchmarking). It is IRT-driven adaptive testing on existing final-answer items, and it reports training-curve monotonicity of 0.99 versus 0.91 for random sampling on HellaSwag, plus fifty times fewer items on MMLU. We run it with the same item budget per model, the same model pool, and the same backtesting protocol. We run it on two item sets: its original suite, and our generated items scored by final answer only. Our proposal beats it if our items show larger spread in fitted ability among the newest model cohort and a smaller held-out rank error at equal item count.

Strongest competitor for capability dimensionality: D-optimal task selection from Certifiable Evaluation. It reports prediction intervals 50% narrower than random selection at nominal 95% coverage, and an effective rank of about 8. We apply the same held-out rank estimator to our final-answer matrix and our step-level matrix, using identical models and items. Beating it means two things: the step-level matrix needs significantly more dimensions to predict held-out cells, and adding those dimensions improves held-out prediction beyond noise.

Step-scoring competitors. Code-structure-style classification reached 81% accuracy against 73% for an LLM judge (Evaluating Intermediate Reasoning). We include an LLM judge grading the same traces, to measure how much the programmatic checker adds. The comparison uses agreement with the checker on a hand-audited subset.

Key ablation. Keep the two-knob grid but score only final answers. This isolates whether the gain comes from the grid design or from step-level scoring.

Why not just…?

Why not just make problems longer or harder on one difficulty scale? A single scale is exactly what we suspect hides structure. The RoPE result shows depth and precision can come apart: positional downscaling improved perplexity while arithmetic exact match stayed near zero. The one-axis version is our final-answer ablation restricted to the diagonal $W \propto D$. If it matches the full grid on held-out rank and backtest discrimination, the second axis is unnecessary, and we would report that.

Why not just run Fluid Benchmarking on existing math sets? It is a real threat on ranking efficiency. Its reported gains are large, and adaptive selection alone may delay saturation. But its items are scored as right or wrong, and the intermediate-reasoning study shows such scores cannot separate sound from brute-force solutions. The direct comparison in Baselines settles which delays saturation more on the newest models.

Why not just grade traces with a strong LLM judge? FaithBench found the best detector at 62.31% balanced accuracy, and found that chain-of-thought judging often did worse than zero-shot judging. An LLM judge also ages along with the models it grades. Executable references remove the judge from the loop. We restrict ourselves to verifiable families for that reason, and we accept that this narrows scope.

Why not just use step-level scoring without the width knob? Then we could not test the RoPE-derived prediction. Without that test, "resolution" would be only a metaphor. The downscaled-model check below would come out null.

Proposed Experiments

We evaluate a pool of open models spanning sizes and release dates, including base models paired with long-context variants extended by position interpolation, plus API models where logging traces is possible. Where they fit the format, we also reuse ToolBench-style tool chains (StepTool) as an out-of-family transfer test. There we treat each tool call as a step, with width equal to the number of arguments.

Check 1: effective rank. On identical items and models, does the held-out effective rank of the step-hazard matrix exceed that of final-answer accuracy? We use the Certifiable Evaluation estimator for both.

Check 2: temporal backtest. We fit item parameters only on models released before a cutoff. We then measure how well each item set separates models released after the cutoff, compared against Fluid-selected items at matched budgets.

Check 3: axis validity via positional interpolation. Downscaled-position variants should show a significantly larger width coefficient $b_m$ than their base models, with little change in the depth coefficient $c_m$. If both change equally, or neither changes, the two knobs are not measuring distinct mechanisms and the proposal's framing fails.

Check 4: splitting as a test of what causes depth attenuation. Divide-and-conquer prompting (When Splitting Makes Stronger) should mainly flatten $c_m$, because it bounds each context slice, while leaving $b_m$ unchanged. Seeing that pattern would show the hazard model explains why known fixes work, not just which models are ahead.