When Does Generator–Verifier Training Converge? A Gradient-Dominance Account

Training a language model to reason by pitting it against a learned checker is becoming a common recipe. One model writes step-by-step solutions. A second model, the verifier, reads them and flags faulty steps. Both are updated with reinforcement learning. The recipe works in practice, but nobody can say when it will converge and when it will oscillate, with the verifier and generator chasing each other in circles. This proposal treats that question as an optimization problem. If the regularization both models already carry (a penalty for drifting from their starting weights) is viewed as curvature in the space of the models' outputs, the game becomes a well-studied kind of min-max problem. Whether training converges then depends on two measurable quantities: how strong that penalty is, and how well-conditioned the map from weights to outputs stays during training.

The payoff is practical. We will measure an empirical "gradient-dominance ratio" along real training runs. If it predicts collapse before collapse happens, it becomes a diagnostic. We will also test whether a simple rule for deciding which model to update next, derived from a Lyapunov function, lets training run with weaker regularization and so reach higher accuracy. The application is math reasoning, where the gaps are well documented. A 7B generator–verifier system reaches only 26.7% on AIME 2024. Small models solve hard problems while failing easier versions of the same problems. Static verifiers cannot tell a convincing flawed proof from a correct one.

Core Claim

A generator–verifier game is hidden bilinear. In the space of output distributions, the payoff is linear in the generator's distribution over solutions and linear in the verifier's flagging probabilities. Bilinear games are the textbook case where simultaneous or alternating gradient play rotates instead of converging.

Two existing results meet here. The hidden convex-concave analysis of Vlatakis-Gkaragkounis et al. shows that a payoff $f(x,y)=g(u(x),v(y))$ is benign when $g$ is strictly convex-concave and the maps $u,v$ keep non-vanishing gradients. Solving Neural Min-Max Games shows that, for two-layer networks, the resulting two-sided Polyak–Łojasiewicz (PL) constant is governed by the smallest singular value of the network Jacobian. The two-sided PL condition says that whenever play is far from equilibrium, at least one player still has a large gradient pointing toward its best response.

The KL penalties used in GRPO-style training (penalties on divergence from a reference policy) are exactly what turns the bilinear $g$ into a strongly convex-concave one. Our working conjecture, to be proved in a simplified setting and tested empirically at scale, is

$$\|\nabla_x f\|^2+\|\nabla_y f\|^2 \;\gtrsim\; \tau\,\sigma_{\min}^2\,\big(\Phi(x,y)-\Phi^*\big),$$

where $x$ and $y$ are the generator and verifier parameters, $\tau$ is the KL coefficient, $\sigma_{\min}$ is the smallest singular value of the parameter-to-output Jacobian on the sampled outputs, and $\Phi-\Phi^*$ is the exploitability, meaning how much either player could gain by switching to a best response. The form comes from composing a $\tau$-strongly convex-concave $g$ with a map whose Jacobian is bounded below by $\sigma_{\min}$.

This yields three predictions. First, training is stable roughly when $\tau\sigma_{\min}^2$ stays bounded away from zero. Second, collapse is preceded by a fall in $\sigma_{\min}$, i.e., rank collapse. Third, an update schedule that drives a gradient-norm Lyapunov function downward should tolerate smaller $\tau$. Smaller $\tau$ matters because heavy KL keeps the generator close to its reference and caps how much reasoning ability training can add.

Relation to Prior Work

RL Tango motivates this work. It co-trains a generator and a generative step-level verifier with interleaved RL and reports roughly doubled AIME accuracy over vanilla GRPO. Its updates are heuristic alternation, and it offers no account of when they cycle or drift. We reuse its setup as the main testbed and do not propose a competing reward design.

The multi-agent reasoning works also motivate the question, and they show that stability is the real bottleneck. The Value of Variance documents debate collapse and repairs it with variance penalties. From Debate to Equilibrium proves a regret bound but only over decoding temperature and repetition penalty, with model weights frozen. Neither analyzes the geometry of weight-space co-training, which is our object.

On the method side, Solving Neural Min-Max Games and Patel et al. supply the Jacobian-conditioning view. Their guarantees cover two-layer networks with $\Omega(n^3)$ width. We do not claim to extend these proofs to transformers. Instead, we measure the quantities those proofs say matter. From Lyapunov Analysis to Algorithm Design supplies the scheduling idea through TALDA, which picks among a gradient step, a role swap, and continued approximation so that a gradient-norm Lyapunov surrogate decreases. TALDA is deterministic and was tested only on quadratics and robust least squares. Adapting it to noisy policy-gradient estimates is part of the work, and it may fail.

Research Program

The first component is a small-scale theory result. Take softmax policies over a finite set of candidate solutions and step-flags, with a feature map whose Jacobian is bounded below. For the KL-regularized zero-sum generator–verifier game in this setting, we will prove that two-sided PL holds with constant proportional to $\tau\sigma_{\min}^2$. We will also prove that alternating updates converge linearly, while at $\tau=0$ they cycle. This is a modest corollary of the hidden convex-concave argument. Its purpose is to make the predictions precise, not to stand as the main contribution.

The second component is a measurement protocol for real runs. The gradient-norm side comes from the policy-gradient estimates already computed during training. Exploitability is estimated periodically. We freeze one player and train a fresh copy of the other against it for a fixed small budget, using LoRA adapters (small low-rank weight updates). The resulting gain is the exploitability estimate.

$\sigma_{\min}$ is approximated by the smallest eigenvalue of the Gram matrix of per-sample log-probability gradients, restricted to the LoRA parameters, computed on a fixed probe batch of sampled solutions. Their ratio gives an empirical PL constant along the trajectory. We also log oscillation directly: the verifier's flag rate and the generator's pass rate over time.

The third component is the intervention: a Lyapunov-gated schedule. At each round, the gate estimates the surrogate $\|\nabla_x f\|^2+c\|\nabla_y f\|^2$ from minibatch gradients. It then chooses whether to update the generator, update the verifier, or keep refining the player whose estimate is currently furthest from a best response, following TALDA's three actions. Variance-reduced gradient estimates and a hysteresis margin keep noise from flipping the decision every step.

The game itself is run in two variants. The answer-anchored variant follows RL Tango, where the verifier is rewarded for agreeing with gold correctness. The executor-anchored variant targets problems without gold answers. There the verifier wins only by producing a concrete counterexample to a flagged step, such as a numerical substitution, that a Python executor confirms. This anchor prevents the degenerate equilibrium in which a verifier with no ground truth simply rejects everything.

Baselines

The naive reference point is single-model GRPO with a rule-based final-answer reward and no verifier. RL Tango reports that it reaches roughly half of Tango's AIME accuracy. It tells us how much of any gain comes from having a verifier at all.

The strongest existing method is RL Tango itself: interleaved GRPO on generator and verifier, with its decayed step-reward schedule. It reports 26.7% on AIME 2024 and 23.3% on AIME 2025 at the 7B/8B scale. We will re-run it on the same base model, data, and GPU-hour budget rather than compare against those reported figures. We give it a hyperparameter search over KL coefficient and the ratio of generator to verifier updates equal in trials to our own search. Beating it means a statistically reliable AIME and Omni-MATH gain at equal compute, or equal accuracy with markedly fewer collapsed seeds. Matching accuracy while being more stable is reported honestly as a stability result, not an accuracy result.

The second strong competitor is optimistic or extragradient descent-ascent applied to the same RL updates. Extrapolating one step ahead is the classical fix for rotation in bilinear games. It addresses the same failure as our gate with no Lyapunov machinery, so it is the most direct threat. It runs under the same budget and search.

The key ablation is a random gate. It keeps the same proportion of generator, verifier, and continue actions as the Lyapunov gate but picks them without looking at gradient norms. If the random gate matches the Lyapunov gate, the benefit comes from the update mix, not the Lyapunov signal.

A static verifier baseline completes the set. A process reward model trained once and frozen is used both as a training reward and for Best-of-$N$ reranking at inference. This tests whether co-training matters at all.

Why not just…?

Why not just turn up the KL coefficient? Under our own account, a larger $\tau$ raises the PL constant and should stabilize training, so this is a real threat. The cost is that a large $\tau$ ties the generator to its reference policy and limits improvement. The decisive experiment is a sweep of $\tau$ for every method. If the gated schedule's accuracy-versus-stability frontier is not strictly better than ungated training's frontier, the gate adds nothing. In that case the useful output is the diagnostic, which tells practitioners the smallest safe $\tau$ without a full sweep.

Why not just use extragradient or optimistic updates, the standard fix for cycling? These may well suffice, which is why they are a headline baseline and not a footnote. Our reason to expect a gap is that their guarantees assume accurate gradients. Policy-gradient noise in long-sequence RL is large, and extrapolation amplifies it. TALDA's gating instead reacts to an aggregate gradient-norm signal that can be averaged over many samples. If extragradient matches the gate at equal compute, we will report that the cheap fix is enough. The diagnostic contribution stands either way.

Why not just train a static process reward model and use Best-of-$N$? The file's evidence points against it at the difficulty levels we care about. Omni-MATH found Best-of-$N$ with static reward models less effective than expected on Olympiad problems. RL Tango reports that static reward models get exploited by the policy and that its co-trained 7B verifier outperformed Qwen2.5-Math-72B-Instruct on ProcessBench. We still include this baseline, because at the GSM level it may be competitive.

Why not just use multi-agent debate at inference time? Debate needs no training, but The Value of Variance reports that standard debate reaches only 68.4% on GSM8K against 92.3% once stabilized by policy optimization. Its cost also grows with the number of rounds. It answers a different question, inference-time aggregation rather than weight-space convergence, so we treat it as context and not as a primary baseline.

Proposed Experiments

All runs use the same 7B base model. Training uses competition-style math problems with gold answers for the anchored variant. The executor-anchored variant drops those answers and keeps only the problem statements.

Evaluation covers AIME 2024 and 2025 and Omni-MATH, where Qwen2.5-MATH-72B-Instruct scores 36.2% according to the Omni-MATH paper. It also covers GSM1k, to check that gains are not memorization; A Careful Examination of LLM Performance on Grade School Arithmetic reports drops of up to 8% for overfit model families. ConsisEval tests hard-to-easy consistency; Can LLMs Always Solve Easy Problems reports that 7B models sit between 50% and 70% there. ProcessBench measures verifier quality; no reported number was found on Lacuna for RL Tango's verifier on ProcessBench, so we will establish it ourselves.

The first decisive check is predictive. Across a grid of KL strengths, learning rates, and five seeds per cell, does a falling empirical PL ratio, driven by $\sigma_{\min}$, precede measured oscillation or verifier collapse? We score this by lead time and by the area under the ROC curve for predicting collapse within the next fixed number of steps. Our account is refuted if collapse occurs while the ratio stays high, or if the ratio carries no more signal than the raw gradient norm.

The second check is the frontier. At each $\tau$, we compare final AIME and Omni-MATH accuracy and the fraction of collapsed seeds across GRPO, RL Tango, extragradient, the random gate, and the Lyapunov gate. The proposal succeeds if the Lyapunov gate reaches higher accuracy at a smaller stable $\tau$ than any baseline, and the random gate does not reproduce this.

The third check is the toy validation. In the finite-solution softmax setting where the theorem applies, the measured PL constant should scale roughly linearly with $\tau\sigma_{\min}^2$. If it does not, the conjecture behind the scaled-up measurements is wrong and we will say so.