Volume-Bounded Policy Warps for Offline Reinforcement Learning

Offline reinforcement learning trains a decision policy from a fixed log of past behavior, with no chance to try new actions. The central failure is extrapolation error: the learned value estimate (the critic) is wrong on actions the data never contains. The policy, which is trained to climb that estimate, finds those errors and exploits them. This note argues that the most damaging form of this exploitation is geometric and can be measured. The policy takes many different plausible actions from the data and squeezes them onto a few spots where the critic is falsely optimistic. In the language of deformations, the map from "what the data would do" to "what the policy does" folds or collapses volume. That shows up as a Jacobian determinant near zero. We propose to build policy improvement as an explicitly invertible warp of the behavior distribution. The warp has a hard floor on how much it can shrink volume and a cap on how far it can move each action. These two numbers, together with the smoothness of the behavior data's density, give a direct bound on how much the policy can over-concentrate relative to the data. That quantity controls how badly extrapolation error can hurt.

In two sentences: sample a behavior action from a generative model of the data, then push it through a learned invertible map that increases value but may not shrink any region by more than a factor $e^{c}$ or move any point more than $\delta$. Because change of variables makes the policy's density exact, these two budgets bound the ratio between policy and data density. That ratio is what extrapolation error feeds on. The budgets replace hand-tuned conservatism with a quantity one can compute and check.

Core Claim

Let $\pi_\beta(a\mid s)$ be the behavior density, meaning the distribution of actions in the data at state $s$. Let $T_s$ be an injective map on the action space, and let the policy be $\pi = T_s\#\pi_\beta$, i.e. sample $a\sim\pi_\beta$ and output $T_s(a)$. Suppose the map satisfies two conditions: $-\log\det J_{T_s}(a)\le c$, so no region shrinks by more than a factor $e^{c}$, and $\|T_s(a)-a\|\le\delta$. Suppose also that the behavior score, the gradient of the log density $\nabla_a\log\pi_\beta$, has norm at most $L_\delta$ within distance $\delta$ of the data. Then change of variables plus a mean-value argument gives, for every action $a'$ the policy can produce,

$$\frac{\pi(a'\mid s)}{\pi_\beta(a'\mid s)} \;\le\; \exp\!\left(c + L_\delta\,\delta\right).$$

The policy therefore cannot place much more mass anywhere than the data does. This holds pointwise, and it is guaranteed by construction rather than encouraged by a penalty.

Here the four directions meet. Topology preservation, meaning injectivity and a positive determinant, is what makes the density formula valid. It also rules out many data modes being funneled onto one spurious critic peak. Score smoothness enters through $L_\delta$, and it tells us where the displacement budget must shrink. The flow-matching error analysis in An Error Analysis of Flow Matching for Deep Generative Modeling shows that velocity and score regularity degrade toward the boundary of the target's support. That is exactly where $L_\delta$ grows, so the safe per-sample step is $\delta(s,a)\propto 1/\|\nabla_a\log\pi_\beta(a\mid s)\|$. This step can be computed from a likelihood-capable behavior flow instead of being tuned by hand. Stable adversarial training supplies the stabilization lens. The actor–critic loop is a two-player game, and $c$ and $\delta$ act as a proximal trust region on it. They play the role that the Jordan–Kinderlehrer–Otto step plays in A Unifying View of Variational Generative Wasserstein Flows, a paper showing that a Wasserstein proximal anchor stops gradient blow-up in adversarial generative training. Our version sets the step size adaptively rather than leaving it as a tuned $\tau$, which that paper names as an open problem.

A second, falsifiable claim comes before the method. In current one-step flow actors, the fraction of noise inputs where the noise-to-action map is nearly singular should predict critic overestimation, measured as the gap between predicted value and Monte Carlo return, across tasks and seeds. If it does not, the geometric story is wrong and the method should not be built.

Relation to Prior Work

The closest work is Continuity-Regularized Flow Matching for Offline Reinforcement Learning (PQL). It identifies "critic-induced deformation": critic gradients shear the velocity field of a flow-matching policy. It counters this with a Frobenius-norm penalty on the velocity Jacobian and with Beta-distributed timestep sampling. PQL motivates our proposal because it found the deformation. It also leaves a precise gap. A Frobenius penalty punishes stretching and shrinking symmetrically. It acts on the intermediate velocity rather than the final map, and it neither forbids collapse nor yields a density-ratio bound. We constrain the log-determinant of the final map, one-sidedly, which is the direction that concentrates mass.

The "obvious" intersection is a hard diffeomorphic policy, for example scaling-and-squaring of a stationary velocity field as in VoxelMorph: A Learning Framework for Deformable Medical Image Registration and Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces. The latter cut folding voxels to an average of 0.2 across 3,731 brain MRIs. We reuse the method: the warp is a composition of small residual steps with a controlled Lipschitz constant, which makes it invertible by construction. We do not adopt the premise. A continuous-time flow-matching ODE with a Lipschitz velocity field is already a diffeomorphism, so "make the flow invertible" adds little to multi-step flow policies. Invertibility alone also bounds nothing about concentration, since an injective map can still shrink a region to near-zero volume. The folding actually occurs in the distilled one-step actors used by fast flow methods, and what matters is the volume floor, not injectivity alone.

Two works attack extrapolation error with generative or transport tools.

Flow Actor-Critic for Offline Reinforcement Learning (FAC) uses an exact-likelihood behavior flow to penalize the critic at low-density actions, and distills a one-step actor. We reuse its likelihood-capable behavior flow to obtain $\pi_\beta$ and its score. FAC constrains where the critic is trusted; we constrain how the policy may redistribute mass. The two are complementary.

Offline Reinforcement Learning with Wasserstein Regularization via Optimal Transport Maps (Q-DOT) replaces unstable adversarial Wasserstein regularization with an input-convex transport map. It reports that the adversarial baseline "AdvW often fails to converge". That negative result constrains us: we avoid any learned adversarial Wasserstein critic. Q-DOT's displacement-style regularization corresponds to our $\delta$ alone, without a volume floor.

Guided Flow Policy (GFP) reweights behavior cloning toward high-value modes. That is orthogonal to our method and can serve as the source distribution.

On theory, Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity shows that explicit pessimism reaches minimax-optimal sample complexity. We will check whether its analysis can be instantiated with our density-ratio bound in place of a coverage assumption. The goal would be a guarantee that is enforced rather than assumed. We present this as a question, not a result.

Research Program

The system has three learned parts. The first is a behavior flow trained by flow matching on the data, used as in FAC for exact $\log\pi_\beta$ and its action gradient. The second is a critic trained with standard temporal-difference targets, where next actions come from the warped policy. The third is a state-conditioned warp $T_s$. It is built as $K$ residual steps $a\mapsto a+h\,v_\theta(s,a)$, with the step's Lipschitz constant kept below one by spectral normalization, so the composition is invertible.

The warp maximizes $\mathbb{E}\,Q(s,T_s(a))$ over behavior samples $a$. It is subject to two per-sample constraints, $-\log\det J_{T_s}(a)\le c$ and $\|T_s(a)-a\|\le\delta(s,a)$, where $\delta(s,a)=\delta_0/(1+\|\nabla_a\log\pi_\beta\|)$. Both are enforced with learned Lagrange multipliers.

Action dimensions in these benchmarks are small. OGBench manipulation has a handful of dimensions and Adroit hands have roughly 24 to 30. At that size the exact Jacobian and its log-determinant cost only a few extra backward passes per sample. The cost PQL pays with Hutchinson estimates therefore disappears, and the constraint is exact rather than stochastic.

Analysis runs alongside. We will prove the ratio bound above and characterize when a single gradient-ascent warp $a+\tau\nabla_aQ$ folds. The warp's Jacobian is $I+\tau\nabla^2_aQ$, so injectivity can fail once the critic's action Hessian has an eigenvalue beyond $1/\tau$ in magnitude. This links critic curvature in unsupported regions directly to collapse. We will also measure it on trained critics.

Baselines

The naive reference is the behavior flow alone with $T_s$ set to the identity. It shows how much any value-driven improvement adds.

The strongest existing method is FAC. It reports an OGBench average of 60.3 against 43.6 for FQL and 31.0 for ReBRAC, and 100% success on puzzle-3x3-play, where baselines "struggled below 30%" (Flow Actor-Critic for Offline Reinforcement Learning). It is the strongest competitor because it already uses exact behavior densities against extrapolation error. The file does not say which OGBench tasks make up that average, so we will re-run FAC with the released settings on our task set, report our reproduction next to the 60.3 figure, and compare only against the reproduction.

GFP is the second reference. It reports 53.2 average over 105 OGBench tasks and 24.5 against FQL's 3.5 on Cube-Triple-Noisy (Guided Flow Policy). It is included because noisy, multi-modal datasets are where collapse should matter most.

PQL is re-implemented from its description. It reports D4RL wins (5 of 6 medium-expert and medium-replay locomotion settings, 11 of 12 Adroit settings) but no numbers in the file, so there is no reported result on Lacuna for its scores. Q-DOT likewise has no reported numbers in the file.

Every method gets the same data, network widths, number of gradient steps, and eight seeds. Every method also gets the same tuning budget: an equal number of hyperparameter configurations, selected on the same held-out tasks. Beating FAC means a higher mean normalized score than our FAC reproduction on the full OGBench set, with non-overlapping 95% bootstrap intervals on the noisy and puzzle subsets, while not losing on D4RL Adroit.

Three ablations isolate the key ingredient. The first uses the same warp with only the displacement budget $\delta$, which approximates a Wasserstein trust region. The second uses only the volume floor $c$. The third applies PQL's Frobenius penalty in place of both. A fourth variant replaces the invertible residual warp with an unconstrained MLP given the same penalties as soft losses, to test whether injectivity by construction matters.

Why not just…?

Why not just use PQL's Jacobian penalty? PQL penalizes the squared Frobenius norm, which grows when the map stretches as well as when it shrinks. It can therefore suppress useful expansion toward high-value regions while allowing a few near-zero singular values, which is exactly collapse. The PQL-penalty ablation on an identical architecture decides this. If it matches our method on the noisy and puzzle tasks, the volume-floor claim is wrong.

Why not just add a stronger behavior-cloning weight or a displacement trust region? This is a serious threat, especially on locomotion, where the data is nearly unimodal and small moves suffice. A displacement bound alone still lets nearby modes merge. Our prediction is that the displacement-only ablation matches the full method on D4RL locomotion and falls behind on multi-modal OGBench data, where OGBench shows performance swinging from 99% to 6% with dataset noise. If displacement-only matches everywhere, the method reduces to a trust region, and we would report that.

Why not just smooth the critic, for example with an action-gradient or Hessian penalty? Our folding analysis says collapse needs large critic curvature, so smoothing the critic attacks the same cause. It is the most plausible competitor. It also blunts the critic everywhere, including where real value differences are sharp, whereas the volume floor constrains only the policy. We will run a critic-Hessian penalty with an equal tuning budget and report both return and value accuracy. If it wins, the right conclusion is that critic curvature control is the mechanism and the warp is unnecessary.

Why not just make the flow policy a hard diffeomorphism? Multi-step flow ODEs already are diffeomorphisms, and injectivity does not bound concentration. The MLP-warp ablation and the diagnostic study below test whether the determinant, rather than invertibility, is what tracks overestimation.

Proposed Experiments

The first experiment is a cheap check that gates the rest. Train FQL-style and FAC-style one-step actors on OGBench puzzle-3x3-play and cube-triple-noisy, the humanoid maze datasets, and D4RL Adroit cloned and expert. For each actor, record two quantities. The first is the fraction of noise inputs with $\log\det J$ of the noise-to-action map below a threshold, together with the largest eigenvalue of the critic's action Hessian at those actions. The second is the gap between predicted Q and Monte Carlo return. A rank correlation near zero across tasks and seeds would falsify the premise.

If the premise survives, the main experiment runs the full method and all baselines on OGBench, D4RL Adroit and Gym-MuJoCo, and Minari locomotion. Metrics are normalized score or success rate, the overestimation gap, and the measured density ratio $\pi/\pi_\beta$ from the exact likelihoods. The decisive outcomes are three. First, the measured ratio must stay under $\exp(c+L_\delta\delta)$, which checks the theory. Second, the overestimation gap should shrink relative to FAC and PQL without a matching loss in return. Third, the gain should concentrate on noisy and multi-modal datasets while vanishing on unimodal locomotion. A final sweep over $c$ traces the trade-off between conservatism and stitching, which OGBench highlights as the regime where over-conservative methods fail.