Problems, Not Proofs: Private Reasoning Curricula for Sub-Billion Models

Organizations increasingly want small language models (under a billion parameters, small enough to run on a phone or an on-premise box) that can do multi-step quantitative reasoning in their own domain: dosing calculations, internal pricing rules, engineering tolerances. Three facts make this hard. First, the obvious recipe of having a large "teacher" model write step-by-step solutions and training the small model to copy them actively harms models this small. Second, the domain problems are private, so they cannot be sent to an external teacher, and the trained model must not leak them. Third, teacher-written solutions carry the teacher's mistakes into the student. We propose that the teacher should write problems and never solutions. The private side releases only a differentially private (DP) histogram, meaning a count table with calibrated noise added so that no single record can be inferred from it, over the abstract structure of its problems: what operations they involve, how many steps they take, and how many unknowns they have. A public teacher then writes fresh, diverse word problems with those structures. A symbolic solver computes every answer. The small model learns from the answers only through reinforcement learning with checkable rewards, and the mix of structures it sees is continually steered toward problems it solves about half the time.

In two sentences: the privacy budget is spent on a low-dimensional summary of problem structure rather than on text, so the noise that privacy requires never touches the logic of any individual problem. The teacher's role is reduced to writing varied problem statements, so its errors in reasoning cannot reach the student, and the student learns by trial and reward, the one regime known to improve 0.5B models.

Core Claim

For sub-billion reasoning students under data-access constraints, a curriculum made of solver-answered problems whose structural distribution is released under DP will beat three alternatives at the same privacy level ε (the standard DP parameter, where smaller means stronger protection). The first is DP-trained text generators that produce whole synthetic problems with answers. The second is teacher-trace distillation. The third is reinforcement learning on public math data. The gain should be largest exactly where the private distribution differs from public data. This combines three findings that have so far only been studied in pairs.

The first finding is that copying teacher solutions fails at this scale. Effective Learning for Small Reasoning Models reports that distillation drops Qwen2.5-0.5B-Instruct from 45.5% to 18.7% on GSM8K and from 6.2% to 3.3% on OlympiadBench. In the same study, Group Relative Policy Optimization (GRPO), a reinforcement-learning method that rewards a model for sampled answers that check out, gives small but stable gains. This motivates using reinforcement learning only.

The second finding is that solver verification removes teacher hallucination from training data. Neuro-Symbolic Data Generation for Math Reasoning mutates formalized problems inside SMT and SymPy solvers to produce 860K verified problems. That work assumes open seed corpora and unrestricted GPT-4 access, and it fine-tunes 7B models with supervised learning. We supply the method's verification step but neither its seed access nor its training regime.

The third finding is that DP synthetic text works for shallow tasks. DistilDP and KnowledgeSG generate synthetic corpora from DP-trained models and evaluate on perplexity, classification, and medical QA. Neither tests whether DP noise breaks multi-step derivations, and that gap is what our structural design is meant to avoid.

The closest pipeline is DIAG, which has a teacher generate follow-up problems at the student's roughly 50% pass-rate boundary. DIAG sends student error traces to an unrestricted teacher, draws on open seed sets, and trains 7–8B students with preference optimization. We keep its competence-boundary idea but show it can be run at zero privacy cost. Because pass rates are measured on synthetic problems, computing them is post-processing under DP and consumes no budget.

Research Program

The pipeline has four stages. In the first stage, which happens on the private side, a locally hosted open model translates each private problem into a SymPy program. A deterministic abstraction then maps that program to a structural signature $s$: the multiset of operation types, the depth of the dependency graph, the number of unknowns, and whether the problem calls for an equation system, a ratio, or a counting step. Problems that fail to formalize go to a residual bin.

In the second stage, the private side releases noisy counts $\tilde h(s) = h(s) + \mathcal{N}(0,\sigma^2)$. Here $h(s)$ is the number of private problems with signature $s$, and $\sigma$ is set by the Gaussian mechanism for a target $(\epsilon,\delta)$. Each problem contributes to exactly one bin, so the sensitivity is 1. Nothing else leaves the private side.

In the third stage, a public teacher receives a signature and writes a new word problem with a new story and new numbers. The teacher also emits a SymPy program for the problem. We keep the problem only if two conditions hold: an independent re-parse of the text yields a program with the same signature, and the solver returns a unique answer. The teacher's natural-language solution, if it writes one, is discarded.

In the fourth stage, the student trains with GRPO on these problems. Each training round samples signatures with weight

$$w(s) \propto \max(\tilde h(s), 0)\, p(s)\,(1-p(s)),$$

where $p(s)$ is the student's current pass rate on synthetic problems of signature $s$. The second factor is largest at a 50% pass rate, where reinforcement learning gets the most signal. It addresses the starvation that DIAG documents for static pools and the near-zero starting accuracy that defeats pure reinforcement learning on very small models.

The analysis focuses on three questions. The first is whether the DP histogram carries useful information beyond a uniform or public prior over signatures, and how that depends on ε and on how far the private distribution lies from public data. The second is whether solver answers, compared with teacher answers, matter more as problem depth grows. The third is how much of the private distribution resists formalization, which is the residual bin that bounds the method.

Baselines

The naive reference is the untouched student: Qwen2.5-0.5B-Instruct and Qwen3-0.6B evaluated zero-shot. Effective Learning for Small Reasoning Models reports 45.5% on GSM8K, 31.4% on MATH500, and 6.2% on OlympiadBench for Qwen2.5-0.5B-Instruct, and 56.4% on MATH500 and 20.2% on OlympiadBench for Qwen3-0.6B. We will re-measure these numbers under our decoding settings.

The strongest data-free competitor is public-only GRPO: the same student trained with GRPO on GSM8K and NuminaMath training problems, with no private information. The same study reports this regime reaching 7.6% on OlympiadBench and 32.4% on MATH500 for Qwen2.5-0.5B-Instruct. We will give it the same number of rollouts, the same number of updates, and the same hyperparameter search. Beating it means a statistically significant gain on the private-distribution test set; matching it on public benchmarks is acceptable.

The strongest privacy competitor is a DP text-generator pipeline in the style of DistilDP. We fine-tune a generator on the private problem-answer pairs with DP-SGD (stochastic gradient descent with clipped, noised gradients) at the same ε. We sample synthetic problems with answers and train the student with GRPO on those answers. A variant of this baseline re-answers the sampled problems with our solver, which isolates whether the benefit comes from structural release or from solver answers. No reported result on Lacuna exists for DP synthetic data applied to multi-step math reasoning, so we must establish this baseline ourselves.

A distillation baseline trains the student on teacher-written solutions to our synthetic problems. We run it in two forms: plain supervised fine-tuning, and REDI's asymmetric objective over correct and incorrect traces (Harnessing Negative Signals). REDI reports 83.1% on MATH-500 at 1.5B, but that figure comes from 131k open traces and a larger student. Two non-private upper references show what privacy costs: GRPO directly on the private training pool, and the same pipeline using exact rather than noisy counts.

Three ablations isolate each ingredient. One replaces the DP histogram with a uniform distribution over signatures. One removes the $p(1-p)$ weighting. One substitutes teacher answers for solver answers.

Why not just…?

Why not just run GRPO on the private problems locally? If the model never leaves the premises and the pool is large, this is a real threat, and it is our non-private upper reference. The target setting is different: the student is shipped to devices or clients, so it must not memorize records. KnowledgeSG reports about 97% name extraction from non-privately fine-tuned models. We will plant canary problems with unique entities and numbers, and measure extraction and membership inference for both approaches. We will also run DP-SGD applied to GRPO itself at matched ε. If DP-GRPO matches our method, the structural route has no advantage, and we will say so.

Why not just train on public math data and ignore the private distribution? This works when the private domain looks like GSM8K. That is why the evaluation includes a deliberately shifted private domain, and why the uniform-signature ablation exists. If the DP histogram adds nothing over public-only GRPO on the shifted domain, the central claim fails.

Why not just distill teacher solutions, which is cheaper? At 0.5B, the cited study shows distillation and supervised fine-tuning degrading accuracy, and it shows chaining distillation into reinforcement learning to be unstable. We include both supervised fine-tuning and REDI on our synthetic problems so that the comparison holds the data fixed and varies only the use of teacher solutions.

Why not just use rule-based templates per signature? Templates guarantee valid problems, but Neuro-Symbolic Data Generation reports that they generalize poorly because the language is repetitive. We test this with a template generator conditioned on the same DP signatures, which isolates the value of the teacher's language diversity.

Proposed Experiments

We simulate privacy in two ways. In the first setting, the GSM8K training set plays the private pool, and evaluation uses the GSM8K test set, SVAMP, and ASDiv. In the second, harder setting, the private pool is a shifted domain: MATH training problems restricted to two subjects (for example, counting and number theory), with those subjects' templates removed from the public pool. Evaluation uses the matching MATH500 subjects, and full MATH500 and OlympiadBench serve as transfer checks. Students are Qwen2.5-0.5B-Instruct and Qwen3-0.6B. The metrics are exact-match accuracy, extraction rates for planted canaries, membership-inference area under the curve, and the fraction of private problems that fall into the residual bin. We sweep ε over 1, 2, 4, and 8, which brackets the ε=2 used by DistilDP.

The decisive checks are four. At ε=2 on the shifted domain, the method must beat both public-only GRPO and the DP text-generator baseline, while canary extraction stays near chance. The gap over the uniform-signature ablation must shrink as the private distribution approaches the public one. Solver answers must beat teacher answers by a margin that widens with signature depth. Finally, if the residual bin exceeds about a third of the private pool, the method's coverage is too narrow for real domains, and that result would be reported as a limit rather than averaged away.