Pricing Rejected Samples: Compute-Accounted Synthesis of Math Reasoning Data

Several recent results claim that small synthetic reasoning datasets beat much larger ones. A strong "teacher" model writes step-by-step solutions to math problems, wrong solutions are thrown away, and a student model is fine-tuned on the survivors. These efficiency claims count only the tokens the student trains on. They leave out the teacher computation spent on solutions that were generated and then discarded. For hard problems that discarded share can dominate: if the teacher solves a problem one time in fifty, each kept solution costs fifty generations.

This proposal adds that bill back in. It asks whether difficulty-focused synthesis is still worth it once every floating-point operation (FLOP) of generation, filtering and training comes out of one fixed budget. It also proposes a simple rule for spending generation compute that treats a problem's solve rate as a price, not only as a measure of difficulty.

Core Claim

DART-Math observed that plain rejection sampling (generate solutions, keep the correct ones) is biased toward easy questions. It showed that giving more generation attempts to problems the teacher often fails produces smaller datasets that beat multi-million-sample ones; Llama3-8B rises from 21.2% to 46.6% on MATH. More Data or Better Data? found similar gains from distillation (+8.96 on MATH from OpenR1-Math-220K). Neither paper reports the inference FLOPs behind its data.

The paper that does enforce a single budget is Compute-Constrained Data Selection. It splits a total budget $K$ into selection and training compute and finds that cheap lexical selection (BM25, a keyword-overlap retrieval score) wins at small and medium budgets. That analysis covers choosing among existing examples, where each candidate costs a fixed amount to score. It does not cover creating examples, where the cost of a kept example depends on how hard its problem is.

Our claim is that once generation cost is priced, the best allocation stops favoring the hardest problems and moves toward problems of intermediate difficulty. There should also be a budget threshold below which training on cheaply filtered existing data beats any synthesis.

The mechanism can be written in one line. Let $p_q$ be the teacher's pass rate on problem $q$, $N_t$ the teacher's parameter count, and $L$ the average solution length in tokens. A forward pass costs about two FLOPs per parameter per token, so each accepted solution costs about $2N_tL/p_q$ FLOPs in expectation.

Now suppose each problem's value to the student grows with diminishing returns as $w_q \log(1+n_q)$, where $n_q$ is the number of accepted solutions and $w_q$ is a difficulty weight. The concave form is an assumption. It borrows the repetition-driven utility decay that Scaling Laws for Data Filtering measured on image-text data.

Maximizing total value under a generation budget, by setting the marginal value per FLOP equal across problems, gives

$$n_q + 1 \;\propto\; w_q\, p_q .$$

With DART's own choice of weight, $w_q \propto 1-p_q$, the allocation becomes proportional to $p_q(1-p_q)$. Budget flows to problems the teacher solves about half the time, not to the ones it almost never solves. The prediction is sharp and could easily be wrong: hard examples might be worth far more than a concave value function allows. Either outcome is informative.

Research Program

The work has three parts.

A cost ledger. We build an accounting harness that charges every stage to the same budget $K = C_{\text{probe}} + C_{\text{synth}} + C_{\text{train}}$. The three terms are:

  • $C_{\text{probe}}$: the few teacher samples per problem used to estimate $p_q$. This step is not free; DART also needs it to measure difficulty.
  • $C_{\text{synth}}$: all teacher generations, including rejected ones.
  • $C_{\text{train}}$: student fine-tuning.

We log both analytic FLOPs and measured GPU-hours. The second matters because batched generation and training use hardware very differently, and the complexity literature warns that idealized counts ignore such effects.

Three competing uses of the same budget. For each budget level we compare:

  • synthesizing new solutions under different allocation rules;
  • selecting a subset of an existing solution pool with a cheap selector;
  • training on more lightly filtered existing math text.

Plotting accuracy against budget for each strategy yields a compute-accounted accuracy frontier.

Amortization. A dataset generated once can train many students. Following the finding in Compute-Constrained Data Selection that gradient-based selection only pays off after reuse across about 20 tasks, we report the number of student training runs needed before each synthesis strategy breaks even with the cheap alternatives.

The theory in Why Less is More (Sometimes) plays a constraining role. It predicts that "keep hard" curation helps only when data is plentiful and the generator is strong, and hurts when data is scarce. It treats curation as free, however. Our experiments test whether pricing generation cost moves that phase boundary, and we expect the move to be toward requiring more data.

Setup. The main student is Qwen2.5-3B, following More Data or Better Data?, with Llama3-8B as a check against DART's published numbers. The teacher is DeepSeekMath-7B-RL, which is DART's generator. QwQ-32B serves as a second, costlier teacher to vary $N_t$. We use DART's query pool.

Evaluation is on MATH, plus BBH and MMLU to detect the out-of-domain losses that Rethinking Data Curation in LLM Training attributes to narrow curation. We sweep at least five budget levels spanning roughly two orders of magnitude, each with three seeds.

Baselines

Uniform rejection sampling (do nothing clever). This is DART's "Uniform" variant: the same number of generation attempts for every problem. We run it under exactly the same $K$ ledger as every other method. It is the reference point any allocation rule must beat.

DART-Math-Hard. Attempts are allocated in proportion to the teacher's fail rate (DART-Math). This is the strongest existing method on this question, and its reported result is 46.6% on MATH for Llama3-8B. The file reports no FLOP or GPU-hour cost for DART-Math, so its compute-accounted standing is unknown. That gap is exactly what we measure.

We regenerate DART-Hard ourselves with the same teacher, the same probe budget, the same student training recipe and the same hyperparameter-tuning budget as our rule. Separately, we confirm that our regenerated data roughly reproduces the published accuracy before trusting the comparison.

Cheap selection from an existing pool. BM25 or random subsampling of OpenR1-Math-220K, which is reported to give +8.96 on MATH in More Data or Better Data?. Following Compute-Constrained Data Selection, BM25 is the Pareto-optimal selector at small budgets. We charge this baseline two ways:

  • as a consumer, who pays nothing for the pool's creation;
  • as a producer, who pays an estimated creation cost.

Both views are reported.

More lightly filtered text. Continued training on textbook-style refined math text, where More Data or Better Data? reports +1.72 on MATH for Math-Cosmo. This represents spending the budget on quantity.

Ablation of the pricing term. Our rule is run with $p_q$ removed from the allocation, keeping the concave value and the difficulty weight. This isolates whether the gain comes from treating the pass rate as a price or merely from diminishing returns.

What counts as beating a baseline. Our rule beats a baseline if it reaches higher MATH accuracy at equal total $K$, averaged over seeds, at a majority of budget levels, with no more than a one-point loss on BBH.

Why not just…?

Why not just use the released DART or OpenR1 datasets, which cost a consumer nothing? For a single consumer this is correct, and we report that view. But someone has to produce the data. Choosing how to spend generation compute on new domains or new teachers is a producer's decision. The amortization count tells a producer how many downstream users justify each strategy.

Why not just spend the whole budget training on more filtered existing data? This is a real threat. BM25's dominance at small budgets in Compute-Constrained Data Selection suggests synthesis may lose outright at low $K$. We expect that to happen and will locate the crossover budget. If synthesis never crosses over within our sweep, that negative result is the main finding.

Why not just keep DART's hard-first allocation and add a FLOP counter? Counting alone tells you whether DART is expensive. It does not tell you what to do instead. The ablation of the pricing term and the direct DART-Hard comparison test whether reallocation matters. If DART-Hard matches our rule at equal $K$, the pricing argument fails and we will say so.

Decisive Checks

The proposal stands or falls on four measurements.

Does DART-Hard lose its advantage over uniform sampling once rejected generations are charged? We compare the two at equal $K$ across the budget sweep.

Does the $p_q(1-p_q)$ allocation beat both at equal $K$? The pricing ablation must also show that the pass-rate term is responsible.

Is there a crossover budget? Below some $K$, cheap selection or more filtered text should win, and moving from the 7B teacher to QwQ-32B (raising $N_t$) should push that crossover higher.

Does the empirical allocation curve peak at intermediate pass rates? If the best-performing allocations instead concentrate on problems with near-zero pass rates, the concave-value assumption is wrong. That would identify hard examples as disproportionately valuable, a finding worth reporting in itself.