A robot moving through a crowd makes two kinds of choices at once. It picks a discrete maneuver, such as yielding, passing on the left, passing on the right, or falling in behind a pedestrian. It also picks continuous settings for that maneuver: how fast, how wide a berth, how long to wait. Reinforcement learning methods that handle this "hybrid" action space usually learn their value estimates with a standard safety device, which is to take the minimum of two independently trained value networks. We argue that this device does not affect all maneuvers equally. The minimum of two noisy estimates falls further below the truth when the noise is larger, and noise is largest for the maneuvers whose outcomes depend on what moving people do next. The aggressive maneuvers therefore look systematically worse than they are, the cautious ones look about right, and the robot learns to yield again and again. This is the well-known "freezing robot problem," and on this account it is partly a statistical artifact of how values are estimated.
The proposal tests that claim directly and then builds the fix it implies. The fix lets each discrete maneuver have its own degree of pessimism, and calibrates that pessimism per maneuver against actual returns on held-out episodes. The same mechanism serves sim-to-real transfer: a small buffer of real-robot episodes can re-tune the per-maneuver pessimism without retraining the policy. This matters because the reality gap is also uneven across maneuvers. Close passes are much more sensitive to latency and friction mismatch than waiting is.
Core Claim
Write the clipped double-Q target in a form that exposes its knob. For two critics $Q_1, Q_2$ evaluated at the next state $s'$, next discrete mode $d'$ and its continuous parameters $x'$, we have $\min(Q_1,Q_2) = \bar Q - \tfrac12|Q_1-Q_2|$, where $\bar Q$ is their mean. We generalize this to a target in which the pessimism weight depends on the mode:
$$y = r + \gamma\left[\bar Q(s',d',x') - \beta_{d'}\,\tfrac12\,|Q_1(s',d',x') - Q_2(s',d',x')|\right],$$
Here $r$ is the reward, $\gamma$ the discount, and $\beta_{d'}\ge 0$ a pessimism weight for mode $d'$. Setting $\beta=1$ for every mode recovers the standard minimum, and setting $\beta=0$ gives the plain average used by Revisiting Discrete Soft Actor-Critic. Under a simple Gaussian error model with per-mode critic error spread $\sigma_d$, the minimum is biased downward by about $\sigma_d/\sqrt{\pi}$, which is the expected minimum of two independent equal-variance Gaussians. Bootstrapping compounds this gap over the horizon. The bias is therefore mode-dependent whenever $\sigma_d$ differs across modes, and the discrete selector, which compares modes against each other, inherits that difference as a preference.
The claim has three parts, and each can be falsified. First, with a hybrid actor-critic using the standard minimum, the gap between estimated and Monte Carlo value will be larger for interaction-heavy modes than for yielding, and the gap will widen with crowd density. Second, that gap predicts how often the robot freezes, even when the state representation is held fixed. Third, calibrating $\beta_d$ per mode reduces freezing without the collision increase that a single global reduction of pessimism causes.
Relation to Prior Work
Two literatures motivate the proposal without meeting. The crowd-navigation line treats freezing as a representation problem. Crowd-Robot Interaction adds self-attention over human-robot and human-human interactions. Intention Aware Robot Crowd Navigation adds a spatio-temporal interaction graph and forecasts pedestrian paths. Both use simple flat or standard action heads, and neither measures value-estimate bias. The value-bias line shows that conservative critics paralyze exploration. Better Exploration with Optimistic Actor Critic names this "pessimistic underexploration" in continuous control. Revisiting Discrete Soft Actor-Critic shows the minimum blinds discrete agents to high-reward actions. These results are confined to static MuJoCo or game benchmarks. We do not dispute that representation matters. Instead, we hold the representation fixed by reusing the interaction-graph encoder and vary only the value target. That isolates a cause the navigation papers never test.
Three papers supply method pieces. A Case for Validation Buffer in Pessimistic Actor-Critic learns a single pessimism weight online from held-out transitions. We make the weight per-mode. We also replace its ensemble-mean proxy with Monte Carlo returns from completed validation episodes, because that paper's proxy is reported to degrade in highly stochastic environments, and crowds are such environments. Controlling Underestimation Bias in RL via Quasi-median Operation reduces bias uniformly with a larger critic ensemble. It is our main uniform competitor. For the policy structure, we follow the prompt-conditioning design of Counter-Empirical Attacking based on Adversarial RL, in which the discrete choice conditions the continuous head. The hierarchical results of Deep Multi-Agent RL with Discrete-Continuous Hybrid Action Spaces indicate this converges faster than predicting parameters for every mode.
Two papers constrain the design. Controlling Underestimation Bias in Constrained RL for Safe Exploration shows that underestimating cost is what drives safety violations. Blanket optimism is therefore dangerous, and every result here must report collisions alongside freezing. CHPO is the closest hybrid-action work: it handles hybrid actions under cost constraints with dual critics and actors. It assumes static obstacles, says nothing about estimation bias, and stays in simulation.
For transfer, Reinforced Grounded Action Transformation corrects the simulator's dynamics for continuous control, evaluated against surrogate "real" simulators. DWA-RL reaches real robots by restricting actions to a discretized window of feasible velocities. We correct neither the simulator nor the action set. We correct only the per-mode pessimism, which is a far smaller object to fit from scarce real data.
Research Program
We first build a hybrid crowd-navigation task on the CrowdNav simulator (vita-epfl/CrowdNav) with unicycle kinematics. It has five discrete modes: follow path, yield-and-wait, pass left, pass right, and follow behind. Each mode carries continuous parameters: speed, lateral offset, and wait duration. The policy is a hybrid soft actor-critic. A shared interaction-graph encoder feeds a softmax mode head, and a Gaussian parameter head is conditioned on the chosen mode's embedding.
The diagnostic phase trains this agent with the standard minimum across crowd densities of 5 to 20 humans. It logs per-mode bias, defined as the critic estimate minus the discounted Monte Carlo return from the same state and mode. It also logs per-mode critic disagreement and freezing events. A freezing event is speed below 0.05 m/s for more than 2 seconds while a collision-free path exists, checked by an oracle planner with access to future pedestrian positions.
The method phase holds out 10% of episodes in a validation buffer and updates each $\beta_d$ by a small gradient step on the squared gap between that mode's mean bias and zero. A floor on $\beta_d$ is enforced by a separate collision-cost critic that is kept fully pessimistic, in line with the cost-underestimation finding of MICE.
The transfer phase asks whether per-mode recalibration on a few real episodes recovers performance that a policy trained in simulation loses. It first uses surrogate realities: injected latency, wheel slip, and switching pedestrians from ORCA to social-force behavior. It then uses a differential-drive robot of the TurtleBot class used by prior work, with roughly 30 to 50 real episodes for recalibration.
Baselines
The naive reference is the same hybrid agent with the standard two-critic minimum, $\beta_d=1$ for all modes. Every other method differs from it in exactly one component, so differences can be attributed.
The strongest existing crowd-navigation method on the benchmark we use is LM-SARL from Crowd-Robot Interaction. It reports 100% success in the invisible-robot setting, against 78% for CADRL, and a 10.59 s time to goal in the visible setting, against 12.29 s for ORCA. We run it with its released code under its own action set. We also report our method under the paper's evaluation protocol, so the numbers are directly comparable. Beating it means matching success while reducing time to goal and freezing events in the visible, dense setting where freezing actually occurs. The 100% invisible result leaves no headroom on success.
On SocNavBench, SocNavBench reports 32/33 success for Social Forces, 40 collisions for SA-CADRL, and 64 for a pedestrian-unaware planner. Beating SA-CADRL's collision count while staying within Social Forces' success is the bar. For the interaction-graph navigator of Intention Aware Robot Crowd Navigation, no absolute success number was found on Lacuna, only a 12% ablation gain from human-human attention. We therefore retrain it under our protocol.
The uniform-bias competitors are the quasi-median target over five critics, following Quasi-median Operation; a single learned global $\beta$, following Validation Buffer; and the plain average, following Revisiting Discrete SAC. No reported result for any of these on hybrid-action or crowd-navigation tasks was found on Lacuna. They are reimplemented on our hybrid agent with the same encoder, the same environment steps (5M), and the same random-search tuning budget of 20 configurations per method. The isolating ablation compares per-mode $\beta_d$ against a single global $\beta$ fit with the identical validation procedure. If the two match, the "per-mode" ingredient is not what helps. As a sanity check on generic hybrid tasks, we run on the DI-Engine Parking task, where CHPO reports 37.63 reward against 20.01 for HPPO-Lag at cost ≤ 1. This checks that mode-specific pessimism does not hurt outside navigation.
Why not just…?
Why not just remove the minimum everywhere, with quasi-median or averaged critics? This is the most serious threat. If the bias gap between modes is driven mainly by a shared offset, a uniform reduction fixes it at lower complexity. Our account predicts otherwise. Any single setting either leaves the noisy passing modes underestimated or makes the well-estimated yield mode, and the collision-adjacent states, overestimated. The ablation in which a global $\beta$ is tuned to minimize freezing should then show more collisions than per-mode $\beta_d$ at equal freezing. If it does not, we drop the per-mode claim and report the diagnostic finding alone.
Why not just improve the representation or add pedestrian prediction, as the navigation literature does? Both cited navigation papers reduce freezing this way, and we use their encoder. The question is whether residual freezing remains with a strong encoder and whether it tracks value bias. The diagnostic regression of freezing rate on the per-mode bias gap, at fixed architecture, answers this directly. A null result would support the representation account.
Why not just discretize the parameters, as DWA-RL does, or add an entropy bonus on the mode head? Discretization keeps the same minimum-of-critics target over a larger discrete set, so it should inherit the same variance-dependent bias. We test this by running the discretized variant with the diagnostic. An entropy bonus forces the robot to try aggressive modes but does not correct their values, so it should raise mode switching without lowering freezing in evaluation, when the robot acts greedily. We include it, tuned under the same budget.
Why not just use heavier domain randomization for transfer? Randomization widens the range of outcomes for exactly the interaction modes. Under our account that widens their critic disagreement and their underestimation, which predicts more freezing on hardware. We compare randomized training against nominal training plus real-episode recalibration of $\beta_d$, at matched simulation budget.
Proposed Experiments
The decisive first experiment is the diagnostic. Across densities, per-mode bias for pass-left, pass-right, and follow-behind should exceed that for yield by a margin that grows with density and correlates with per-mode critic disagreement. Freezing rate should then rise with that gap across seeds. If biases are mode-uniform, the project narrows to evaluating uniform corrections.
The second experiment compares per-mode calibration with the global-$\beta$ ablation and the quasi-median target on CrowdNav visible-setting episodes at 10, 15, and 20 humans. The metrics are success, time to goal, collisions, intrusions into personal zones, freezing events, and residual per-mode bias. These are reported over 10 seeds and 500 test episodes each, and the claim is judged on the freezing-versus-collisions frontier, not on any single metric. SocNavBench replays of the ETH and UCY pedestrian datasets then test against non-reactive real trajectories.
The transfer experiment trains in nominal simulation, measures the drop under each surrogate reality and on the physical robot, and then recalibrates only $\beta_d$ from 30 to 50 target episodes. It compares this against retraining with grounded action transformation of the RGAT type on the continuous head, and against domain randomization. Success means that recalibrating five scalars recovers most of the lost time to goal without added collisions. It should also show that the learned real-world $\beta_d$ shift most for the close-passing modes, as the uneven reality gap predicts.
References
- Revisiting Discrete Soft Actor-Critic
- Crowd-Robot Interaction: Crowd-Aware Robot Navigation With Attention-Based Deep Reinforcement Learning
- Intention Aware Robot Crowd Navigation with Attention-Based Interaction Graph
- Better Exploration with Optimistic Actor Critic
- A Case for Validation Buffer in Pessimistic Actor-Critic
- Controlling Underestimation Bias in Reinforcement Learning via Quasi-median Operation
- Counter-Empirical Attacking Based on Adversarial Reinforcement Learning for Time-Relevant Scoring System
- Deep Multi-Agent Reinforcement Learning with Discrete-Continuous Hybrid Action Spaces
- Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration
- CHPO: Constrained Hybrid-action Policy Optimization for Reinforcement Learning
- Reinforced Grounded Action Transformation for Sim-to-Real Transfer
- DWA-RL: Dynamically Feasible Deep Reinforcement Learning Policy for Robot Navigation among Mobile Obstacles
- SocNavBench: A Grounded Simulation Testing Framework for Evaluating Social Navigation