Spending Synthetic Transitions Where the Value Function Bends

Diffusion models can now write convincing fake experience for reinforcement learning. These are generators that learn to turn random noise into samples, here into transition tuples of state, action, reward and next state. SynthER, for example, enlarges an offline dataset with millions of such tuples, and the extra data lets larger networks train without overfitting. These generators spread their samples evenly over the data distribution, and that is expensive. SynthER needs about 90 seconds per 100,000 samples, and a multi-task trajectory diffuser needs 1.9 seconds per trajectory. Much of that work is likely wasted. In parts of the state-action space where the value of acting changes slowly, one real transition already tells the learner about everything nearby, so synthetic neighbours add almost nothing. Generated data is only needed where the value or the dynamics change quickly.

We propose to measure that rate of change and use it to set the generation budget. The rate of change is the local Lipschitz constant, the largest factor by which a small move in input changes the output. We build an adaptive partition of the data, similar to the ball-splitting trees of adaptive-discretization reinforcement learning, and estimate a Lipschitz constant in each cell from a value ensemble and the dynamics. A diffusion model conditioned on the cell then generates only enough transitions to cover the gap between what the real data covers and what that local constant requires. The central claim is that this allocation matches uniform synthesis on D4RL with a small fraction of the generated samples. The claim stands only if the gain comes from the smoothness estimate itself. Generating fewer samples, sampling where data is sparse, and sampling where the critic is uncertain must all fall short of it.

Core Claim and What Is New

Adaptive discretization in continuous reinforcement learning already uses Lipschitz smoothness to argue that uniform coverage is wasteful. AdaQL splits a metric ball once it has been visited often enough and reaches optimal reward on Oil Discovery with about 150 cells, against about 850 for a uniform grid. ADAMB carries the same idea into model-based learning. Both methods motivate this proposal, and both depend on online interaction to fill the cells they create. Neither works with a fixed dataset, and neither uses a generator.

On the generator side, three papers sample without any notion of where samples are needed. SynthER samples uniformly. MTDIFF samples uniformly across 50 tasks. RTDiff generates trajectories in reverse, from unseen states back into the data. It cuts them off with a Mahalanobis out-of-distribution test, a statistical distance check, which bounds plausibility but not usefulness.

ADCM is the nearest use of local sensitivity to cut generative compute. It uses the network Jacobian to place denoising steps along the one-dimensional noise schedule and saves up to 75% of training compute. It adapts along the noise schedule, not over the state-action space. Its savings make each sample cheaper. Ours reduce the number of samples needed. The two should therefore stack, and we test whether they do.

What is new here is a budget rule that ties the number of synthetic transitions in a region to the local smoothness of the value function and the dynamics there. A second piece is an explicit test for cells where smoothness fails.

Research Program

We first fit a value ensemble on the real data, using the IQL or TD3+BC critic trained briefly, together with the diffusion model. On a learned low-dimensional embedding of state-action pairs, we grow a tree in the spirit of AdaQL. A cell $c$ splits when the estimated value change across it, $\hat L_c \cdot \mathrm{diam}(c)$, exceeds a tolerance $\varepsilon$. Here $\hat L_c$ is the larger of two estimates: the maximum finite-difference ratio of ensemble-mean values over pairs of real points in the cell, and the same ratio computed for next states. The embedding is needed because a grid over the raw 20-plus dimensions of a locomotion task would face the curse of dimensionality that AdaQL itself warns about.

Each leaf then receives a target count of transitions:

$$n_c^{*} = \left\lceil \left(\frac{\hat L_c\,\mathrm{diam}(c)}{\varepsilon}\right)^{k_c} \right\rceil .$$

Here $k_c$ is the estimated intrinsic dimension of the data in the cell. The formula is the number of balls of radius $\varepsilon/\hat L_c$ needed to cover a $k_c$-dimensional cell, which is the covering argument behind adaptive-discretization regret bounds. The synthetic budget for the cell is $\max(0, n_c^{*} - m_c)$, where $m_c$ is the number of real transitions already in it. The total is rescaled to a fixed global budget.

Generation must actually save compute rather than discard samples, so the SynthER diffusion model is made conditional on the leaf through classifier-free guidance. Classifier-free guidance trains the same model with and without the condition and then mixes the two predictions at sampling time. With it, every denoising pass produces a sample in the intended cell, and rejection is not needed. Training cost stays close to SynthER's, since the network and data are the same.

The theory already says where this will break: in places where smoothness itself fails. Fan and Ramadge require Lipschitz dynamics for their contraction argument, and LIFT (shortcut augmentation; Trajectory-Level Data Augmentation for Offline Reinforcement Learning) states that its guarantees fail where the value function is not Lipschitz. Both results constrain the design, and our method has to detect these cells. A cell whose $\hat L_c$ fails to shrink, or grows roughly like $1/\mathrm{diam}(c)$, over two successive splits looks like a jump, not a steep slope, and we flag it. For flagged cells we compare two policies. The capping policy assigns no synthetic samples and leaves those regions to real data and the offline method's conservatism. The saturating policy gives them a large share of the budget, in the spirit of DISCO's choice to concentrate generation on ambiguous boundary instances (DISCO). Which policy wins resolves, at least empirically, the open question of whether synthetic capacity belongs on smooth regions, where guarantees hold, or on boundaries, where the most is at stake.

Baselines

The naive reference point is the offline learner (IQL and TD3+BC) trained only on real data with the same network size and number of gradient steps. It sets the floor that any augmentation must clear.

The strongest competitor is SynthER with its uniform unconditional EDM. It is the method our allocation rule directly modifies, and it has the broadest evidence. On Walker2d-Medium-Expert it reaches near-original performance from 3% of the data, larger networks gain 11.7% with its upsampling, and it costs about 90 seconds per 100k samples (SynthER). We found no per-task D4RL normalized scores for SynthER on Lacuna, so we will reproduce them with its released configuration. We run it at its default budget and at matched budgets of 5%, 10%, 25% and 50%, with the same diffusion architecture, training steps, downstream learner and tuning sweep as our method. Beating it means matching its full-budget score, within the seed confidence interval, at 25% of the generated samples or fewer, and outperforming it at every matched budget.

On Maze2d-Large and Kitchen-Partial, which have walls and contact and so give the discontinuity test real force, we also compare against RTDiff, which reports gains of 14.3 and 13.6 points on IQL (RTDiff). Beating it there means a comparable gain at a lower generation budget.

The key ablation isolates the smoothness signal. It keeps the tree, the conditional generator and the per-cell budgets, but randomly permutes the $\hat L_c$ values across leaves. If shuffled allocation does as well as ours, the gain comes from conditional generation or from the partition, not from smoothness.

Why not just…?

Why not just generate fewer samples uniformly? If SynthER's benefit saturates early, a smaller uniform budget would capture most of the savings at no cost in complexity. This is a real threat. The matched-budget SynthER curves test it directly. Our claim survives only if adaptive allocation dominates the uniform curve at low budgets, not merely at the default.

Why not prioritize where the critic ensemble disagrees, or where data is sparse? Uncertainty-driven and inverse-density allocation are simpler and already common. They differ from our rule in one specific way. They push samples into sparse but flat regions, where one real point already settles the value, and they under-serve dense but steep regions, where nearby points disagree in value even though each is well estimated. We run both, ensemble-disagreement allocation and inverse-density allocation, under identical budgets and generator. A diagnostic on a 2D Oil-Discovery-style offline task with a known reward surface and an inserted cliff shows where each rule puts its samples relative to the true local Lipschitz constant. If disagreement allocation matches ours on D4RL, the honest conclusion is that the Lipschitz estimate adds nothing over uncertainty, and we will report it that way.

Why not just add Gaussian noise in flat cells, or anywhere? SynthER found that additive and dynamics noise produce transitions that break the physics. Plain noise is therefore a weak baseline, and we include it only for completeness. The hybrid version is a fair question, though. In cells our estimator certifies as flat, noise of scale $\varepsilon/\hat L_c$ might be enough and would cost nothing. We test it as a variant. If it holds up, it further cuts diffusion calls and strengthens the claim that smoothness, not generator fidelity, decides where expensive samples are needed.

Why not just make each sample cheaper with a consistency model and ADCM? That attacks cost per sample, not the number of samples, so it is complementary to this work. We distill the generator into a one-to-two-step consistency model and run both allocations with it. The savings should multiply. If adaptive allocation loses its advantage once samples are cheap, that is a meaningful limit on the work, and we will report it.

Proposed Experiments

Validation of the estimator comes first. On the 2D Oil Discovery and ambulance-relocation environments used by AdaQL and ADAMB, converted to offline datasets collected by a scripted policy, the true local Lipschitz constant is computable. On these tasks we measure the rank correlation between $\hat L_c$ and the truth, and the precision and recall of the discontinuity flag around an inserted reward cliff.

The main study uses D4RL locomotion at medium, medium-replay and medium-expert levels, plus Maze2d-Large and Kitchen-Partial, with IQL and TD3+BC as downstream learners. It reports normalized return against the number of generated samples and against generation GPU-seconds, with tree construction and estimation time counted honestly, across 5 seeds. A secondary run on V-D4RL, in SynthER's latent space, checks whether the embedding-based tree survives pixel inputs; SynthER reports a 9.5% gain for DrQ+BC there.

Three results decide the work. Adaptive allocation must reach full-budget SynthER performance at a quarter of the budget or less. It must beat shuffled, disagreement-based and inverse-density allocation at matched budgets. On Maze2d and Kitchen, the capping and saturating policies for flagged cells must separate clearly, which tells us whether synthetic data near discontinuities helps or hurts. If none of these hold, the Lipschitz assumption is not a useful guide to generative budgets in these benchmarks, and that negative result is worth stating plainly.