Engineers routinely hand-write curricula for hard reinforcement learning tasks. A peg-insertion robot first learns to reach the peg, then to grasp it, then to align it. Each of these early stages has a goal simple enough that a person can check by eye whether the learned behavior is sensible. The usual practice is to use the stages only to order training and then throw the stage policies away. We propose keeping them. The places the stage policies visit, taken together, form a library of behavior that a human has approved. The final agent can then be trained on a dense, hackable shaping reward, which is what makes it learn quickly, while being penalized for spending time in states and actions that no approved stage ever reached. The curriculum stops being a training schedule and becomes a safety boundary.
The technical vehicle is occupancy measure regularization. An occupancy measure is the discounted frequency with which a policy visits each state-action pair. Preventing Reward Hacking with Occupancy Measure Regularization (ORPO) showed that penalizing the divergence between an agent's occupancy measure and that of a single trusted reference policy prevents reward hacking far better than the usual per-action KL penalty. That paper's main limitation is the reference itself. If the reference never visits the states the task requires, the penalty caps performance. Our claim is that the union of curriculum-stage occupancies covers the states of a full multi-stage task even when no single stage policy does. Regularizing toward the best mixture of stage occupancies should therefore keep ORPO's protection against hacking without its performance ceiling.
Core Claim
Let $q_\pi$ be the agent's occupancy measure, and let $q_1,\dots,q_k$ be the occupancy measures of $k$ human-approved curriculum-stage policies. The agent maximizes
$$\max_\pi \; J_{\text{proxy}}(\pi) \;-\; \lambda \min_{w \in \Delta_k} D_{\chi^2}\Big(q_\pi \,\Big\|\, \textstyle\sum_{i=1}^k w_i q_i\Big),$$
where $J_{\text{proxy}}$ is the expected return under the dense shaping reward, $\Delta_k$ is the set of nonnegative weights summing to one, and $\lambda$ sets the penalty strength.
Two facts about occupancy measures make this objective sensible. First, as laid out in Geometry and convergence of natural policy gradient methods, the set of valid occupancy measures is convex. Any mixture of stage occupancies is therefore itself a realizable visitation pattern, and normalizing it per state yields a policy that blends the stages according to how often each one visits that state. Second, the $\chi^2$ divergence is convex in its second argument. Once density ratios are estimated, the inner minimization over $w$ is a small convex problem.
The $\chi^2$ penalty grows sharply wherever the agent visits pairs outside the mixture's support. In practice, then, the objective says: exploit the shaping reward freely, but stay inside the union of approved behavior. The union matters because each stage covers a different segment of the task, from approach to grasp to alignment.
Why This Is Different From Prior Work
Three lines of work motivate the proposal and differ from it in specific ways. ORPO supplies the core method, a $\chi^2$ occupancy penalty estimated with a discriminator, but it uses one fixed reference and no curriculum. We change what the agent is anchored to, not how the anchor is enforced.
Curriculum Reinforcement Learning for Complex Reward Functions is the closest work on the curriculum side. It trains on a simplified reward and then switches to the full constrained reward, and it reuses Phase 1 transitions through a flexible replay buffer. It operates purely in policy and value space and offers no protection against gaming the simplified reward. It also reports destabilization at the phase switch. We do not switch rewards at all. Stage data enters only through the regularizer, so there is no phase transition to destabilize.
Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency (RFCL) avoids dense rewards entirely because they invite cheating. It relies on demonstrations and on resetting the simulator to arbitrary demonstration states. We take the opposite bet: keep the dense reward for its speed and neutralize its loopholes with the occupancy boundary. Our stages are defined by sub-goals from the ordinary initial-state distribution, so no resets are needed.
Defining and Characterizing Reward Hacking constrains our ambitions. It proves that over all stationary policies, any non-trivial proxy is hackable, and that unhackability becomes possible only on restricted policy sets. We do not claim to make the shaping reward unhackable. We claim to restrict optimization to a region a human has inspected, which is the practical version of the restriction that theorem says is necessary.
Self-Paced Deep Reinforcement Learning automates curriculum pacing but assumes the reward is faithful. We treat manual curricula as useful precisely because a human has looked at each stage.
Research Program
The method has three phases.
In the first phase, the human writes a short curriculum, typically three to five sub-goal tasks. Each stage policy is trained with the same off-policy learner, and its rollouts are spot-checked visually or against a sparse success flag. The stage replay buffers are kept.
In the second phase, a single multi-class classifier is trained to distinguish agent transitions from transitions drawn from each of the $k$ stage buffers. Its outputs give all ratios $q_i/q_\pi$ at once. The ratio to any mixture is then available in closed form, so the weights $w$ can be re-optimized every few thousand steps by projected gradient descent without retraining the classifier.
This design targets the sample-cost gap that ORPO leaves open. The reference side of the classifier is trained entirely on stage data that already exists. Only the agent side needs fresh rollouts. All stage-training environment steps are charged to our method's budget.
In the third phase, the target-task agent is trained with soft actor-critic on the dense proxy reward minus the penalty, with $\lambda$ chosen on a held-out task.
Alongside the method, we will run a coverage analysis that tests the mechanism directly. For successful target-task trajectories collected from any method, we measure the fraction of their state-action pairs that fall within the classifier-estimated support of each single stage and of the union. If the union-support story is right, union coverage should be high while every single-stage coverage is low, and this gap should predict which methods succeed. We will also compute the Effort of Sequential Learning from Studying Exploration in RL: An Optimal Transport Analysis of Occupancy Measure Trajectories, which measures how long the agent's learning path through occupancy space is relative to the direct route. The question is whether the boundary straightens learning or merely blocks it.
Baselines
The naive reference point is soft actor-critic trained on the dense shaping reward with no regularizer, plus the same agent trained on the sparse true reward alone. Together they bracket the problem: the first should learn fast and hack, and the second should be safe but slow or stuck. The standard manual-curriculum practice is a third reference. Here the agent is fine-tuned sequentially through the stages and then on the target with the dense reward, with no occupancy term. It shows whether the curriculum alone already prevents hacking.
The strongest existing competitor on the hacking question is ORPO with a single reference. We run it in two variants: anchored to the final, most capable stage policy, and anchored to the best single stage chosen in hindsight. ORPO is the strongest competitor because it is the only cited method shown to beat action-level KL regularization on long-horizon hacking. The file gives a number for it only on language-model fine-tuning: a 52.83% AlpacaFarm win rate against 51.50% for the KL penalty. For its sequential-control domains, no reported result was found on Lacuna for ORPO. We will therefore reimplement it with the same classifier architecture, discriminator update schedule, and $\lambda$ search as our method.
The action-level KL penalty toward the stage policies is included as the regularizer most practitioners would reach for. The two-stage reward curriculum of Curriculum Reinforcement Learning for Complex Reward Functions is run on its own modified DeepMind Control tasks. The file reports no numeric success rates for it, only that it learns constrained finger spin where baselines fail completely, so it too will be reimplemented.
On ManiSkill2 we also report RFCL as a reference. It uses 5 demonstrations plus state resets and reports about 39.4% success on PlugCharger where RLPD gets 0%. This is not a like-for-like competitor. Our method uses no demonstrations and no resets. Matching or approaching that figure would show that a cheap manual curriculum can stand in for demonstrations. Falling well short would locate the limit of the approach.
Two ablations isolate the key ingredient. The first fixes the mixture weights to be proportional to stage data volume, which amounts to regularizing toward pooled stage data. The second removes one stage that covers a critical sub-task. All methods share the same environment-step budget, with stage-training steps included, the same learner and network sizes, five seeds, and an equal hyperparameter search of twenty configurations each.
Beating the baselines means two things at once. On true success, the method must show a statistically significant improvement over the best single-reference ORPO run. On the gap between proxy return and true return, it must stay no worse than single-reference ORPO, while the unregularized dense-reward agent shows a large gap.
Why not just…?
Why not just anneal ORPO's penalty strength toward zero so the agent eventually escapes a weak reference? Annealing loosens the boundary everywhere at once, including in the hacked directions. The cyclic-loop exploits documented in AI Safety Gridworlds are exactly what a slowly loosened boundary admits. Our method loosens the boundary only where approved stages actually went. We include annealed single-reference ORPO as an extra baseline. If it matches our true success without a larger proxy-true gap, that would undercut the claim, and we will report it as such.
Why not just regularize toward a behavior-cloned policy, or toward the pooled replay data of all stages? This is the most serious threat, because a pooled reference is a fixed-weight mixture of stage occupancies and already has the union support. We treat it as our first ablation, not as a strawman. If fixed weights perform as well as optimized weights, the contribution narrows to the claim that curriculum-stage data is the right reference. That claim is still untested in the literature and still distinct from ORPO's single trained reference. Optimized weights should matter most when stages differ greatly in data volume, so we will deliberately vary stage buffer sizes to test it.
Why not just use the manual curriculum the normal way and skip the regularizer? Sequential fine-tuning carries no memory of where approved behavior lived once the agent reaches the dense-reward target stage. Curriculum Reinforcement Learning for Complex Reward Functions reports degenerate spinning or stopping behavior under composite rewards, along with instability when phases switch. The sequential-curriculum baseline directly measures whether the curriculum alone suffices.
Why not just use the sparse true reward with a good learner and avoid shaping entirely? That is RFCL's position, and it works when demonstrations and resets are available. Without them, RFCL reports that RLPD with 5 demonstrations stays at 0% on PlugCharger. The sparse-only baseline shows how much the shaping reward is buying.
Proposed Experiments
We use three test beds, each with a proxy reward the agent optimizes and a separate true measure it never sees.
The first is ManiSkill2 PlugCharger and PegInsertionSide. The proxy is the dense shaped reward that RFCL rejected as cheat-prone. The true measure is the sparse success flag. The curriculum stages are reach, grasp, lift, and align.
The second is the Boat Race and side-effects environments from AI Safety Gridworlds. Here the hidden performance function $R^*$ is the true measure, and the stages are partial-course sub-goals. These small environments allow occupancy measures to be computed exactly, so the classifier's ratio estimates can be checked against ground truth.
The third is the constrained DeepMind Control tasks and mobile robot navigation from the two-stage reward curriculum paper. The proxy is the unconstrained task reward, and the true measure is success subject to the constraints.
The metrics are true success or $R^*$, the proxy-true gap, total environment steps to reach 50% of final true success, and support coverage.
Three checks are decisive.
The first is the comparison with single-reference ORPO. Our method must reach higher true success, and the coverage analysis must show that this advantage coincides with union coverage that no single stage provides.
The second is a negative control. When the alignment stage is deleted, true success on the insertion tasks should collapse toward the single-reference level. If it does not, the gains come from something other than the stage boundary, and the mechanism claim is wrong.
The third is a hacking stress test. In Boat Race, and in a ManiSkill variant where the shaping reward contains a deliberately planted contact bonus, our method must keep the proxy-true gap near that of single-reference ORPO, while the unregularized and sequential-curriculum agents exploit the loophole.
References
- Preventing Reward Hacking with Occupancy Measure Regularization
- Geometry and convergence of natural policy gradient methods
- Curriculum Reinforcement Learning for Complex Reward Functions
- Reverse Forward Curriculum Learning for Extreme Sample and Demo Efficiency
- Defining and Characterizing Reward Gaming
- Self-Paced Deep Reinforcement Learning
- Studying Exploration in RL: An Optimal Transport Analysis of Occupancy Measure Trajectories
- AI Safety Gridworlds