When a language model answers a hard math question, a common way to buy accuracy is to sample many solutions and take a majority vote. The share of samples that agree is then used as a confidence score, and sampling stops once agreement is high. This proposal starts from a simple suspicion: agreement is overconfident because the votes are not independent. Sixty samples that all run the same algebraic manipulation and reach the same answer are closer to one piece of evidence than to sixty. We propose to count how many genuinely different solution strategies stand behind an answer. We would do this with a clustering method that infers the number of clusters from the data instead of fixing it in advance. That count, rather than the raw vote share, would drive both the confidence the model reports and the decision to stop, resample, or refine.
The opportunity follows from three findings that have not been put together. Test-time sampling saturates because rollouts collapse into a few redundant reasoning clusters (Refining Over Resampling). Majority agreement and answer entropy are miscalibrated predictors of whether more compute will help: they overlap between problems about to recover and problems about to collapse (AERA). And how many strategies a problem admits clearly varies from problem to problem, so any fixed number of clusters will merge strategies on hard problems and split trivial rewordings on easy ones. This is the classic fixed-cardinality failure that non-parametric models were built to avoid.
Core Claim
We claim that, conditional on the same majority vote share, an answer supported by several distinct strategy clusters is more likely to be correct than one supported by a single cluster. We further claim that a non-parametric mixture over the model's own hidden-state summaries of its reasoning traces provides two things that surface statistics lack. The first is an estimate of how many independent strategies back each answer. The second is an online estimate of the probability that the next sample would open a strategy not yet seen. Together these give a confidence score that should be better calibrated than vote share, and a stopping rule that spends compute only where a new strategy is still plausibly reachable.
The mechanism uses a Dirichlet process mixture. This is a Gaussian mixture whose number of occupied components is inferred, fitted here with a truncated stick-breaking variational approximation of the kind used in GMDI. It is fitted per problem over embeddings of the $n$ traces sampled so far. Under the Chinese restaurant process view of this prior, the probability that the next trace starts a new cluster is
$$p_{\text{new}} = \frac{\alpha}{n + \alpha},$$
where $\alpha$ is the concentration parameter, which we infer per problem under a vague prior rather than fixing it. The confidence in answer $a$ counts clusters rather than samples:
$$c(a) = \frac{\sum_{k} w_k \, \mathbf{1}[a_k = a]}{\sum_{k} w_k}.$$
Here $k$ indexes occupied clusters, $a_k$ is the majority answer within cluster $k$, and $w_k$ is a saturating function of the cluster's posterior mass, so that a large cluster counts only somewhat more than a small one. The raw $c(a)$ and $p_{\text{new}}$ are then passed through the same held-out isotonic calibration map that every baseline receives.
Relation to Prior Work
Two lines of work motivate the proposal by showing the gap. Refining Over Resampling compares breadth, meaning resampling $N$ paths, against depth, meaning $D$ rounds of self-critique and correction. It documents diversity saturation and reports large compute-normalized gains from depth, but it uses uniform $N$ and $D$ for every problem and never models how many reasoning clusters exist. AERA trains a stopping gate on 55 checkpoint features to predict whether more compute will fix or break an answer. It shows that consensus statistics are poor proxies, but its features are answer strings and scalar agreement statistics, not the latent structure of the traces. We keep AERA's framing of stopping as prospective utility. What we change is the evidence unit, which becomes the strategy cluster.
Two works supply the method. The stick-breaking Gaussian mixture of GMDI provides the inference machinery. GMDI clusters domains; we cluster reasoning traces within a single question, a setting where the true count ranges from one to many. PROBECAL shows that hidden states carry correctness information that final logits hide, cutting ECE on TabMWP from 0.690 to 0.035 with a small MLP probe. We therefore embed traces using the generating model's own mean-pooled hidden states rather than an external sentence encoder. PROBECAL treats candidate traces as isolated items and trains a supervised probe. We instead exploit the relations among traces, and our signal needs no training beyond a monotone calibration map.
Other works constrain the design. Fact-Level Confidence Calibration and Self-Correction finds that small models remain fully confident in their own hallucinations. This warns that anything built from a single trace's self-assessment will fail. Our signal comes from disagreement across traces, so it degrades differently: a model that is confidently wrong in only one way produces a single cluster, which our score treats as weak evidence. Large Language Models Must Be Taught to Know What They Don't Know shows that verbalized confidence is unstable, which rules out asking the model how sure it is as the baseline to beat. PACE models internal embeddings as a Gaussian mixture over concepts and names fixed $K$ as its open limitation. That paper is evidence that fixed-$K$ mixtures over transformer states are a recognized weak point, though it concerns vision explanation rather than reasoning.
Research Program
We would build a sequential controller around the per-problem mixture. After each batch of samples it refits the mixture, which is cheap because $n \le 64$ points in a low-dimensional projection of the hidden states. It then reads off $c(a)$ for the leading answer and $p_{\text{new}}$, and picks one of three actions. It stops when calibrated confidence is high and $p_{\text{new}}$ is low. It resamples when $p_{\text{new}}$ is high, since unexplored strategies may still exist. It switches to depth refinement in the style of Refining Over Resampling when the traces have collapsed into one or two clusters but confidence remains low. That last case is where breadth has saturated and only critique can add information.
The analysis stage asks whether the clusters mean anything. For 100 MATH500 problems we would have strategy labels assigned by a strong model under a written rubric, with human spot checks, and measure agreement with the mixture's partitions using the adjusted Rand index. We would also measure how the inferred cluster count varies with the dataset's difficulty level. If the clusters track wording rather than method, we would try alternative embeddings: later layers, pooling over only the equation-bearing tokens, or an external math-aware encoder. We would report which one works rather than claim the first did.
Baselines
The naive reference point is plain majority vote at a fixed budget of 64 samples, with vote share as confidence. It tells us what calibration and accuracy cost when nothing adaptive is done. Adaptive-Consistency (ASC), which stops once majority agreement crosses a threshold, is the obvious cheap stopping rule. AERA's reported evidence that it misses collapses makes it the first bar any consensus-based controller must clear.
The strongest existing method is the AERA residual-utility gate. AERA reports 92.61% accuracy on GSM8K with Qwen2.5-7B, against 93.01% at a fixed 128-sample budget, while cutting completion tokens by 95.99%. On GPQA Diamond it reports beating ASC by over 32 points on the subset of questions that collapse. We would reimplement it on the same Qwen2.5-7B samples, with the same calibration split and the same threshold-tuning budget as our controller. Beating it means matching its GSM8K accuracy at equal or lower token cost while achieving lower ECE on the reported confidence. On GPQA Diamond it means at least matching its accuracy on the collapse subset.
For depth refinement, the reference is the verifier-free breadth-plus-depth procedure of Refining Over Resampling. It reports 58.0% on MATH500 with Qwen2.5-1.5B, against 29.6% for the strongest verifier-based baseline. We would run it at its uniform $N$ and $D$ and compare accuracy at matched average tokens. For calibration there is no reported result found on Lacuna for the ECE of majority vote, ASC, AERA, or refine-over-resample confidence on MATH500, GSM8K, or GPQA Diamond. We therefore compute all of these ourselves under a shared isotonic calibration map.
The ablations isolate each ingredient. Answer-only clustering tests whether strategy clusters add anything beyond counting distinct final answers. Fixed-$K$ k-means with $K$ chosen by BIC per problem tests whether non-parametric inference matters. Agglomerative clustering with a tuned distance threshold, combined with a Good-Turing estimate of the unseen fraction, tests whether a simpler rule gives the same count and the same new-strategy estimate. Adding our two features to AERA's gate tests whether they are complementary to its feature set or redundant with it.
Why not just…?
Why not just count distinct final answers, or use answer entropy? In math, answer equivalence gives a free and exact clustering, so this is the most dangerous alternative. The core claim is specifically that strategy diversity carries information that answer agreement does not. The decisive check below tests this directly. If, at fixed vote share, accuracy does not differ between answers backed by one strategy cluster and answers backed by several, the proposal fails. We would report that outcome as a negative result rather than rescue it.
Why not just use fixed-$K$ or threshold clustering instead of a Dirichlet process? This is a real threat. A well-tuned distance threshold may recover the same partitions, and BIC over $K$ is what many practitioners reach for. Evidence from other settings favors inferred cardinality: GMDI beat a single-Gaussian prior on DG-60 (99.3% vs. 95.9%), and TSM reached a $P_k$ of 0.6% against 10.9% for Bayesseg on Choi's dataset. Neither result concerns reasoning traces, though. Our specific argument is that a threshold rule gives a partition but no principled online probability of a new strategy, and that the right threshold should vary with problem difficulty. If the threshold-plus-Good-Turing ablation matches the Dirichlet process on both ECE and tokens-to-accuracy, we would recommend the simpler method and keep the strategy-counting claim as the contribution.
Why not just train a probe on hidden states, as PROBECAL does, or use AERA's gate? A supervised probe needs labeled traces from the target distribution, and AERA itself notes sensitivity to the small calibration set. Our signal needs only a monotone calibration map. It is not obvious that ours wins in-distribution, and we expect AERA may. The fair test has two parts. First, we train each method on GSM8K and evaluate on GPQA Diamond, where transfer is hard. Second, we add our features to the gate and check whether calibration improves. Complementarity would justify the method even if the standalone gate is stronger.
Why not just use a bigger model? Humanity's Last Exam reports GPT-4o at an 89% RMS calibration error. Calibrated Language Model Fine-Tuning reports that more accurate transformers produce sharper, more overconfident probabilities. Scale therefore does not remove the need for an evidence-counting signal. We would still run Qwen2.5-1.5B and 7B to see whether the effective-strategy effect weakens with scale.
Proposed Experiments
We would use GSM8K, MATH500, AMC, and GPQA Diamond with Qwen2.5-1.5B and Qwen2.5-7B, matching the setups of AERA and Refining Over Resampling. We would sample 64 traces per problem, cache their hidden states, and replay all controllers offline on identical samples so that differences come only from the decision rule. Metrics are accuracy, completion tokens, ECE and AUROC of the reported confidence, and, following AERA, the recovery and collapse counts on GPQA Diamond. Calibration maps and thresholds are fitted on a disjoint split drawn from the training portion of MATH and from GSM8K. They are frozen before test.
The first decisive check is the stratified analysis. We bin problems by majority vote share, then compare accuracy between answers supported by one strategy cluster and by several. A consistent gap across bins is the effect the whole proposal rests on. The second check asks whether $p_{\text{new}}$ predicts whether additional samples actually yield a new cluster, which would show the stopping signal is itself calibrated. The third check is the accuracy-versus-tokens curve of the full controller against AERA, ASC, and uniform refine-over-resample. The fourth is the cross-benchmark transfer test, where the absence of a trained probe should matter most. Positive results on the first two checks, together with lower ECE than vote share at equal tokens, would support the claim. Failure on the first check would end it regardless of how the other checks turn out.
References
- Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning
- AERA: Adaptive Evidence Residual Allocation for Efficient Test-Time Reasoning
- Bayesian Domain Adaptation with Gaussian Mixture Domain-Indexing
- Uncertainty Calibration for Tool-Using Language Agents
- Fact-Level Confidence Calibration and Self-Correction
- Large Language Models Must Be Taught to Know What They Don’t Know
- Probabilistic Conceptual Explainers: Trustworthy Conceptual Explanations for Vision Foundation Models
- A benchmark of expert-level academic questions to assess AI capabilities
- Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data