Many control problems ask an agent to make two kinds of choice at once: a discrete one (which actuator to fire, which parking curve to follow, which lane maneuver to start) and a continuous one (how far, how hard, at what angle). In a safety-critical setting the agent also has to keep an expected safety cost under a budget. The natural off-policy recipe is to learn a reward estimate and a cost estimate for every discrete option, discard the options whose estimated cost is too high, and pick the best remaining one. This recipe has a hidden flaw. The options that survive the cost filter are disproportionately the ones whose cost the learned estimator happened to underestimate. This is a "winner's curse" on feasibility, and it gets worse as the number of discrete options grows. The agent then trusts actions that are less safe than it believes, and those errors propagate through the learned values.
We propose to fix this the way Double Q-learning fixed reward overestimation. One set of reward and cost estimators decides which discrete options look feasible and which looks best. A second, independently trained set evaluates the chosen option's reward and cost for learning targets and for the safety mask. We build this into an off-policy, sample-efficient learner for the constrained hybrid-action setting. That setting currently has only an on-policy method, CHPO. We test the specific prediction that the benefit grows with the number of discrete options and vanishes when there is only one.
Core Claim
Write a hybrid action as a pair $(d, x)$: a discrete choice $d$ from a set $D$, plus continuous parameters $x$ for that choice. This is the parameterized-action form formalized in Extending the Bellman equation for MDPs to continuous actions and used as the constrained hybrid-action problem in CHPO. An off-policy safe learner with a parameter actor $x_d = \mu(s,d)$, a reward critic $Q_r$ and a cost critic $Q_c$ forms its next-state target from
$$d^\ast = \arg\max_{d \,:\, Q_c(s,d,x_d) \le \kappa(s)} Q_r(s,d,x_d),$$
where $\kappa(s)$ is the cost the agent may still spend from state $s$.
When $Q_c$ is noisy, the constraint inside the argmax selects branches partly because their cost noise is negative. The expected gap $\mathbb{E}[Q_c(s,d^\ast,x_{d^\ast})] - c^{\pi}(s,d^\ast,x_{d^\ast})$ between the estimate and the true cost-to-go $c^\pi$ is therefore negative. Its magnitude grows with the number of near-feasible branches, for the same order-statistics reason that drives reward overestimation in Double Q-learning. Two further steps make it worse. The reward argmax on top of the filter adds ordinary overestimation. The continuous actor, trained against the same critics, adds a per-branch version of both biases.
Our estimator, which we call decoupled feasible selection, keeps two independently trained critic pairs $(Q_r^A, Q_c^A)$ and $(Q_r^B, Q_c^B)$. Pair A chooses the feasible set and $d^\ast$. Pair B supplies the reward and cost values used in the target, and the cost value used by the safety mask. The roles swap on alternate updates. This is not the same as taking the minimum of twin reward critics and the maximum of twin cost critics. That simpler fix applies a uniform pessimism that does not depend on how many branches competed. Decoupling removes the bias that the selection step itself introduces.
Research Program
The learner has three parts, each drawn from existing work.
The first part is the value machinery. It is a hierarchical hybrid actor-critic: a categorical head over $D$ and one continuous parameter head per branch. This follows the structure that Deep MAHHQN showed trains faster than flat multi-pass hybrid Q-learning. The cross-fitted critic pairs described above sit on top of it. Both critic pairs are trained off-policy from one replay buffer, but on independent minibatches and from independent initializations, so their errors are approximately independent. This addresses the open limitation the CHPO authors state: their method is strictly on-policy.
The second part turns the trajectory-level cost budget into the per-state allowance $\kappa(s)$. We use the backward value functions of Satija et al., which estimate the cost already spent and subtract it from the budget. That paper solved discrete actions with a linear program and continuous actions with a first-order safety layer, but treated them as separate problems. We use both in one hybrid step. A discrete mask removes every branch whose pair-B cost estimate exceeds $\kappa(s)$. The renormalized categorical distribution then samples among the survivors, as in the rule shield of PPO-RS. Within the chosen branch, a one-step linearized projection of $x_d$ against $Q_c^B$ bounds the continuous parameters. This keeps the shield cheap, avoiding the online optimization latency that Realizable Continuous-Space Shields reports. It also avoids solving a mixed-integer program at every step. Unlike PPO-RS, the mask comes from a learned, decoupled estimate rather than from rules extracted after crashes.
The third part addresses data near the safety boundary. FDPI showed that a safe agent stops seeing violations, so its cost critic degrades late in training. FDPI's remedy is a reckless "dual" policy that deliberately probes the boundary. In a hybrid space, that remedy can flip the discrete mode, for example from brake to accelerate, which is the most dangerous possible probe. We restrict probing to the continuous parameters of branches that have already passed the mask. We direct it toward states where $Q_c^A$ and $Q_c^B$ disagree most, because disagreement is a free signal of where cost data is missing. We treat this as a secondary contribution and ablate it separately.
The analysis contributes a short bias statement. For $k$ near-feasible branches with independent cost-estimate noise of scale $\sigma$, single-estimator feasibility selection underestimates the selected branch's cost by an amount that grows roughly like $\sigma\sqrt{2\log k}$, which is the scaling of the expected minimum of $k$ Gaussians. Decoupled evaluation has zero selection bias when the two estimators' errors are independent. We will measure this bias directly rather than only assume it.
Baselines
The naive reference point is an unconstrained hybrid learner with a fixed reward penalty for cost: off-policy hybrid actor-critic, tuned on the penalty weight. The research file's evidence, from Quanyi Li et al. and CHPO's unconstrained ablation, predicts that it is either unsafe or useless. We include it only to calibrate the scale of rewards and costs.
The strongest existing method is CHPO. It is an on-policy hybrid actor-critic that switches each update between cost minimization and reward maximization depending on whether the budget is exceeded. It is the only method designed for exactly this problem, and it reports a Parking reward of 37.63 against 20.01 for HPPO-Lag while keeping cost at or below 1. No open implementation is listed, so we will reimplement it. We treat reproducing roughly 37.63 on Parking as a gate before making any comparison. We will run it on the same DI-engine hybrid suite and cost threshold, with the same hyperparameter-search budget measured in configurations tried. We will compare at both equal environment steps and convergence. Beating CHPO means matching its final cost compliance with comparable or higher reward using clearly fewer environment steps, or achieving higher reward at the same number of steps, with fewer cumulative training-time violations.
Among off-policy competitors, the strongest is a hybrid soft actor-critic with Lagrangian cost handling and twin critics. It takes the minimum of two reward critics and the maximum of two cost critics, built on the same hierarchical actor as our method. This is the pessimistic-ensemble alternative. It isolates whether decoupling matters, as opposed to merely having two critics. We also run PADDPG-Lag, as in CHPO, and a single-critic version of our learner.
The key ablation keeps everything identical, including two critic pairs and the same number of parameters and updates. It switches only the target and mask from "select with A, evaluate with B" to "select and evaluate with the same pair." A second ablation removes the disagreement-directed probing.
Why not just…?
Why not just take the maximum of twin cost critics, as safe off-policy methods commonly do? This is the most serious competitor, and it may close much of the gap when $D$ is small. The difference is that its correction is the same regardless of how many branches compete. A pessimism level tuned for few branches undercorrects when there are many branches, and one tuned for many branches overcorrects when there are few. Overcorrection is the "safe but useless" failure. The HardMove sweep described below settles the question. If max-of-twins matches our method's reward at equal cost across all branch counts, we will report that decoupling adds nothing beyond pessimism.
Why not just stay on-policy with CHPO? On-policy cost estimates do not carry a selection bias through bootstrapped targets in the same way. The cost is sample complexity, which the CHPO authors themselves name as the obstacle to physical deployment. If off-policy decoupled learning does not reach CHPO's compliance with substantially fewer samples, the proposal fails on its own terms. The steps-to-threshold comparison measures this directly.
Why not just discretize the continuous parameters and use Double DQN with a discrete action mask? This would give decoupled selection and PPO-RS-style masking for free. However, the grid grows multiplicatively with the number of parameters and the bin resolution, which is the problem described in Breaking the Grid. Coarse bins also give the jerky control that the driving and robotics literature in Direction 3 warns about. We run this baseline on HardMove and Parking at three bin resolutions to show where it breaks. Our own analysis also predicts that the winner's curse worsens with more discrete options, so we expect a fine grid to suffer from it.
Why not just relax the discrete choice into a continuous variable and use FOCOPS or SAC-Lagrangian? A relaxed mode variable has no meaning at intermediate values, and the hierarchical-versus-flat results of Deep MAHHQN argue against flattening. We include a Gumbel-softmax SAC-Lagrangian run on Parking as a sanity check rather than as a main competitor.
Proposed Experiments
The primary testbed is the DI-engine hybrid safety suite from CHPO. Moving and Sliding involve navigating around danger zones, Sliding with inertia. HardMove fires $n$ on/off actuators with continuous distances. Parking chooses a curve type and a travel distance. All four use the episode-cost threshold of 1. We report final reward, final episode cost, cumulative training-time violations, and environment steps until the cost first stays under threshold.
The decisive experiment sweeps HardMove's $n$ so that the number of discrete options grows from a handful into the hundreds. At each $n$, we measure the per-branch cost-estimation bias by comparing critic estimates against Monte Carlo rollouts from replayed states. Our claim is supported only if three things hold together as $n$ grows. The single-pair learner's cost underestimation on selected branches must grow. Our learner's must stay near zero. And the reward gap at equal cost between our method and max-of-twins must widen.
The matching falsification check uses Safety-Gymnasium navigation tasks from Safety Gymnasium, where there is effectively one discrete branch. There, decoupled feasible selection should reduce to ordinary twin critics and show no advantage. A gain there would mean our improvement comes from something other than the mechanism we claim.
For the exploration component, we track the fraction of near-boundary samples in the replay buffer late in training. FDPI reports that this fraction drops below 1% without dual exploration, and that dual exploration keeps it about ten times higher. We will check whether disagreement-directed parameter probing preserves boundary data without increasing training violations, compared against an FDPI-style dual policy that is allowed to switch discrete branches.
As a transfer check, we add continuous parameters (target speed and lateral offset) to highway-env's discrete maneuvers. We compare against PPO-RS's rule mask to test whether a learned, decoupled mask avoids the collisions that its rule database must first experience in order to learn.
References
- CHPO: Constrained Hybrid-action Policy Optimization for Reinforcement Learning
- Extending the Bellman equation for MDPs to continuous actions and cont. time in the discounted case
- Double Q-learning
- Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces
- Constrained Markov Decision Processes via Backward Value Functions
- A Rule-based Shield: Accumulating Safety Rules from Catastrophic Action Effects
- Realizable Continuous-Space Shields for Safe Reinforcement Learning
- Breaking Safety Paradox with Feasible Dual Policy Iteration
- MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning
- Breaking the Grid: Distance-Guided Reinforcement Learning in Large Discrete and Hybrid Action Spaces
- Safety Gymnasium: A Unified Safe Reinforcement Learning Benchmark