Training Chunk Critics for the Choice, Not the Value

Some of the strongest recent offline reinforcement learning agents work in two steps. First, a generative policy proposes several candidate action chunks, meaning short sequences of future actions such as the next five motor commands. Second, a learned critic, a network that scores how much future reward each chunk will bring, picks the highest-scoring one. The critic is trained like an ordinary regression model: it is penalized for any error in its value estimate, anywhere. But the agent only uses the critic to take a maximum over a handful of candidates. An error that leaves the ranking unchanged costs nothing. A small error that lifts a bad chunk above a good one costs a lot. The best-of-N step also actively looks for whichever candidate the critic overrates most. This is the predict-then-optimize gap, the mismatch between how accurately a model predicts and how good the decisions built on those predictions are, and it sits in the middle of chunked reinforcement learning.

We propose to train the chunk critic partly with a decision-focused loss, one that directly penalizes selection regret over the sampled candidate set. Choosing one of N candidates is a linear program over the probability simplex, the set of weight vectors over candidates. That makes the Smart Predict-then-Optimize surrogate loss exact and cheap here: no solver needs to be differentiated, because the "solver" is an argmax. The claim is specific and testable. At matched compute, a chunk critic trained with a regret-aware auxiliary loss should pick better chunks and succeed more often on long-horizon manipulation than the same critic trained only with temporal-difference regression. The gain should come from fewer ranking errors, not from lower value error.

Where this sits in prior work

The mechanism comes from chunked reinforcement learning. Reinforcement Learning with Action Chunking (Q-chunking, QC) trains a flow-matching generator of action chunks and a chunked critic, then acts by sampling candidates and keeping the one the critic scores highest. Scalable Offline Model-Based RL with Action Chunks (MAC) adds a dynamics model that jumps a whole chunk ahead in one prediction, $(s_t, a_{t:t+k-1}) \mapsto s_{t+k}$, and keeps the same rejection-sampling selection. Both train their critics by squared temporal-difference error. Neither treats selection as a decision problem whose regret is the real target. We keep their generator and selection rule unchanged and change only what the critic is trained to get right.

The sequence-modeling side supplies the critic architecture. Chunking the Critic (T-SAC) uses a small causal transformer that reads a state and an action sequence and outputs a value for every prefix of that sequence. We reuse this design so the critic scores chunks token by token. T-SAC keeps a step-by-step actor and uses the transformer only for more stable multi-step value targets; it never selects among candidates.

The decision-focused learning side supplies the loss and also limits how it can be used. Smart "Predict, then Optimize" introduces the convex surrogate loss SPO+ for linear objectives. Melding the Data-Decisions Pipeline shows that decision-focused training helps most when the model cannot fit everything, which is the regime a critic faces on out-of-distribution candidate chunks. Decision-Focused Learning to Predict Action Costs for Planning is the closest precedent. It applies SPO+ to predicted action costs inside a symbolic planner and lowers Transport regret from about 5.8% to 4.2%. However, its actions are discrete primitives with fixed costs, and there is no learned generator or bootstrapped value. Learning MDPs from Features differentiates through a full MDP solution, a Markov decision process that fixes the states, actions, rewards and transitions, and needs low-rank Hessian approximations to be tractable. Chunk selection sidesteps that cost entirely. Two papers constrain the design. Decision-Focused Model-based Reinforcement Learning for Reward Transfer shows that decision-focused models become brittle when the objective shifts. Decision-Focused Learning via Tangent-Space Projection of Prediction Error notes that regret-optimal predictions can be statistically distorted. A critic's values are reused as bootstrap targets for training, so distorting them is dangerous. This is why the decision loss goes on a separate head, as described below.

Core Claim

The critic produces scores $\hat Q_1,\dots,\hat Q_N$ for the N candidate chunks drawn from the generator at state $s$. Each candidate also gets a reference value $y_j$, the best available estimate of its true return. Let $j^\star = \arg\max_j y_j$ be the candidate with the highest reference value. Writing SPO+ for maximization over the simplex gives

$$\ell_{\text{SPO+}} = \max_j \big(2\hat Q_j - y_j\big) - 2\hat Q_{j^\star} + y_{j^\star}.$$

This loss is zero when the critic already ranks $j^\star$ first with enough margin. Otherwise its gradient raises $\hat Q_{j^\star}$ and lowers the score of the candidate most likely to be wrongly chosen, with a margin equal to that candidate's true value shortfall. Put simply, the loss is a regret-weighted hinge on the argmax, and nothing else in the training signal plays that role.

The loss needs reference values for candidates the dataset never executed. We obtain them from a MAC-style chunk dynamics model: $y_j = \hat r_j + \gamma^k V(\hat s^{\,j}_{t+k})$. Here $\hat r_j$ is the model-predicted reward over the chunk, $\gamma$ is the discount factor, $k$ is the chunk length, $\hat s^{\,j}_{t+k}$ is the model's predicted state after executing chunk $j$, and $V$ is the current bootstrapped value function.

The critic has two heads on a shared transformer trunk. A value head is trained only by temporal-difference regression and supplies all bootstrap targets. A selection head is trained by temporal-difference loss plus $\lambda\,\ell_{\text{SPO+}}$ and is used only at action time. Keeping the heads apart confines the distortion that decision-focused training causes to the one place it helps. It also lets us measure that distortion directly.

Research Program

The work has three stages.

Build. We start from the QC and MAC recipes: a flow-matching chunk generator, a chunk dynamics model, and a transformer chunk critic in the T-SAC style. We add the selection head and the loss above and change nothing else.

Measure the gap before closing it. OGBench runs in MuJoCo, where the simulator state can be saved and restored. On a fixed set of evaluation states we therefore execute every sampled candidate, continue with the same policy until the episode ends, and repeat this over several seeds. This gives Monte Carlo estimates of true candidate values. From them we compute three quantities as N grows: selection regret, the critic's rank correlation with true values, and the critic's squared error. The central diagnostic is whether selection regret tracks critic error or drifts away from it. The predict-then-optimize framing predicts drift, because larger N gives the argmax more chances to exploit errors.

Close the gap and test where it matters. Here we measure whether the decision-focused head reduces regret and raises task success. We also check whether the benefit depends on the quality of the model-generated reference values, by swapping in simulator-true labels on a small subset as an oracle. A secondary extension treats the pair (candidate, number of chunk steps to execute before replanning) as the decision. The critic's prefix values define that larger choice set, so the same loss applies. This addresses the tension over reactivity raised by QC and by T-SAC's rejection of actor-side chunking. We treat it as exploratory, not as part of the main claim.

Baselines

The naive reference point is the chunked agent with N=1: take the generator's first sample with no critic selection. It shows how much selection contributes at all.

The strongest existing method is QC, run from the same codebase with the same generator, data, chunk length, critic architecture, gradient steps, and a hyperparameter search of equal size. The file reports its numbers. QC reaches roughly 86% aggregated success across five OGBench domains, compared with 50–60% for prior methods such as FQL, and 74% on cube-quadruple, where most one-step temporal-difference baselines score 0% (Reinforcement Learning with Action Chunking). Beating QC means a statistically reliable gain over 74% on cube-quadruple in our reproduction, measured against our own rerun of QC rather than the quoted figure. It also means no regression on the other domains.

MAC is the model-based competitor. Because our method uses its dynamics model, MAC controls for the possibility that the model alone explains any gain. MAC reports 99% on puzzle-4x5 (Scalable Offline Model-Based RL with Action Chunks). That task is at the ceiling, so we use it only to check for regressions.

Four ablations isolate the key ingredient. The first applies the same model-labeled targets with a squared-error loss instead of SPO+, which tests whether the benefit comes only from extra labels. The second uses a regret-free pairwise ranking loss on the same labels. The third uses a single head, so the decision loss also shapes the bootstrap targets. The fourth replaces model labels with simulator-true labels as an upper bound. Every variant gets the same number of model calls and gradient steps.

Why not just…?

Why not just sample fewer candidates, or make the critic bigger? Fewer candidates reduce exploitation of critic errors but give up the selection benefit that drives QC's gains. A bigger critic lowers average error, and SPO+ results under misspecification (Smart "Predict, then Optimize") suggest that lower average error does not guarantee correct ranking on out-of-distribution chunks. The N-sweep and critic-width sweep in the measurement stage settle this directly. If regret already falls to near zero as critic size grows, the proposal has no problem to solve, and we will report that.

Why not just select pessimistically with a critic ensemble? Picking by an ensemble's lower confidence bound is the standard fix for overestimation, and it is a real threat to this proposal. It attacks the same failure without labels or a model. We will run ensemble lower-confidence selection at matched parameter count and also in combination with the decision-focused head. If pessimism alone matches the regret reduction, the honest conclusion is that the gap is mostly a variance problem, and the contribution narrows to the measurement methodology.

Why not just use the dynamics model to score candidates at test time? Scoring each candidate by its model-predicted return removes the critic's error, but it inherits the model's error. MAC reports that chunk models struggle in contact-rich settings such as HumanoidMaze. It also adds N model calls per decision. We include this as a baseline. If it wins on accuracy, the decision-focused critic still has to justify itself on latency, and we will say so.

Why not just use a plain ranking loss? If pairwise ranking matches SPO+, then the regret weighting adds nothing, and the simpler loss should be preferred. The ranking ablation exists to decide this question.

Proposed Experiments

The main suite covers the OGBench manipulation domains used by QC (cube-double, cube-triple, cube-quadruple, scene, and puzzle-3x3 and 4x4) with 64 or 128 candidates, eight seeds, and success rate as the primary metric (OGBench). The secondary metric is Monte Carlo selection regret on 500 saved states per task.

Three checks decide the outcome. First, in the untreated QC agent, selection regret must grow with N while critic squared error stays flat. Without this pattern the gap does not exist in this setting. Second, the SPO+ selection head must lower selection regret more than the squared-error-on-model-labels and pairwise ranking ablations. Third, the regret reduction must translate into higher success on cube-quadruple than QC's 74%, and the value head's bootstrap error must stay unchanged relative to QC.

The oracle-label run shows how much of any shortfall is caused by errors in the chunk model's reference values. Robomimic tool_hang, which appears in the file with no reported chunked-RL number on Lacuna, serves as a transfer check on human-teleoperated data.