Robots that learn to walk or drive among people are mostly trained in simulation and then deployed directly, with no further learning on the real robot. The usual trick for making that work is domain randomization: vary the simulator's parameters during training so the real world looks like one more variation. In crowd navigation, though, the most important unknown is not friction or mass. It is how people respond when the robot moves toward them. Do they yield, ignore it, or stop? Current simulators either hard-code one reaction rule or replay pedestrian paths that ignore the robot. The obvious fix is to train pedestrian agents with reinforcement learning so they react to the robot. That fix is expensive: each time the robot's policy changes, the pedestrian policies have to be retrained. This nested structure is the same bottleneck that makes inverse reinforcement learning slow.
We propose to make the pedestrians' response cheap enough to randomize. Each simulated pedestrian chooses its next velocity from a small discrete set, using a short-horizon, temperature-controlled softmax over a cost made of goal progress, personal space, and willingness to yield to the robot. This response is computed in closed form at every step, so no pedestrian is ever trained. Domain randomization then runs over the cost weights, the temperature, and the reaction delay, and therefore over how reactive the crowd is, not only over where people walk. The prior for these parameters is fit to recorded human trajectories in a single pass, using the inverse action-value method of Deep Inverse Q-learning with Constraints. That method needs discrete actions, which is a limitation for torque control but a natural fit for pedestrian velocity choices. The claim is that a robot trained against this family transfers better to unseen reaction styles, and freezes less, than robots trained against fixed rule-based crowds, uniformly randomized crowds, or adversarially learned crowds, at a small fraction of the compute of learned-follower schemes.
Core Claim
Three properties have been treated separately. Crowd navigation work showed that reactivity matters: Relational Graph Learning for Crowd Navigation argues that treating humans as non-reactive produces the "freezing robot problem", where the planner decides every path is unsafe and stops. Sim-to-real work for dynamic obstacles, such as Omni-Perception, randomizes obstacle velocities but lets obstacles follow fixed, non-reactive trajectories, because modeling interaction online was too costly. Game-theoretic work such as Oracles & Followers treats reactive agents as followers in a leader-follower game. That framing is correct, but the follower is retrained for every leader update.
Our position is that pedestrian reaction does not need a learned follower. A closed-form, soft-optimal responder is expressive enough to span the range of real reactions, and cheap enough to sample from widely. The idea of collapsing an inner optimization into an analytic step comes from linearly-solvable control (Inverse Optimal Control with Linearly-Solvable MDPs) and from Boltzmann-policy inversions (Deep Inverse Q-learning with Constraints). We reuse that idea for the simulated environment, not for the learner.
Concretely, pedestrian $i$ at joint state $s$ picks velocity $v$ from a discrete set $V$ (for example 5 speeds by 16 headings) with probability
$$\pi_i(v \mid s) \propto \exp\big(\beta_i \, Q_{w_i}^{H}(s, v)\big).$$
Here $\beta_i$ is a rationality temperature. $w_i$ weights the cost terms: goal progress, distance to other pedestrians, distance to the robot, and a yield bonus for giving way along the robot's predicted path. $Q_{w_i}^{H}$ is an $H$-step soft lookahead value. It is computed by enumerating $V$ while holding the other agents at constant velocity, which keeps the cost per pedestrian per step at roughly $|V| \cdot H$ cost evaluations. A reaction delay $d_i$ makes pedestrian $i$ respond to the robot state from $d_i$ steps earlier. The randomized parameter vector is $\xi = (w, \beta, d, H)$ per pedestrian, drawn from a crowd-level distribution $p_\phi(\xi)$.
Setting the robot-avoidance and yield weights to zero recovers non-reactive walkers. Large weights with high $\beta$ produce highly deferential crowds. Mixtures within one episode produce the heterogeneous crowds that a single rule-based model cannot represent.
Research Program
The work has three parts.
Responder simulator. We implement the responder inside the CrowdNav simulator (CrowdNav Codebase), keeping its circle- and square-crossing scenarios and its robot action space so that published policies can be used without modification. We measure wall-clock time per step against ORCA and against learned pedestrian policies.
Data-anchored prior. We fit the prior $p_\phi$ from recorded pedestrian trajectories. The recordings are discretized onto $V$, and inverse action-value iteration recovers per-trajectory $Q$-values from the empirical action frequencies. A least-squares projection onto the cost features then gives a $(w, \beta)$ estimate for each recorded person. The set of estimates defines an empirical prior.
This step addresses a known weakness. IQ-Learn observes that rewards recovered by value-based inversion depend on the training dynamics and do not transfer cleanly. Our fitted quantities are cost weights inside a responder whose dynamics we control, not a transferred reward. If the projection residual is large, that tells us the responder family is too narrow, and we will report it.
Curriculum over reaction parameters. We expand the prior with an entropy-maximizing curriculum in the style of Domain Randomization via Entropy Maximization. It maximizes the breadth of $p_\phi$ subject to the robot's success rate staying above a threshold $\alpha$. We use DORAEMON's procedure as published, but over the new parameter space. Changing the curriculum algorithm is not part of the claim. This keeps the contribution isolated: what we vary is which environment dimensions get randomized and how cheaply reactive dimensions can be simulated.
The robot policy architecture is held fixed across all conditions. We use the attention-based value network of Crowd-Robot Interaction (LM-SARL), and repeat the main comparison with the relational-graph planner of Relational Graph Learning for Crowd Navigation. This separates the effect of the training crowd from the effect of the policy.
Baselines
The naive reference point is the default CrowdNav training setup: the robot is trained against ORCA pedestrians with fixed parameters. This is how LM-SARL and MP-RGL were trained. In the invisible-robot circle-crossing setting with 5 humans, Relational Graph Learning for Crowd Navigation reports 96% success, 2% collisions, and 1.86 s extra time for MP-RGL, against 90% success for LM-SARL and 43% for ORCA itself. These numbers come from a setting in which humans do not react to the robot, and are evaluated on the same pedestrian model used in training. They therefore measure in-distribution performance. We reproduce them first, as a check on our implementation, and then evaluate the same trained policies on held-out reaction styles.
The strongest simple competitor is uniform randomization over the parameters of existing pedestrian models. This means ORCA's neighbor distance, time horizon, preferred speed, and radius, plus open-loop trajectory perturbations in the manner of Omni-Perception. ORCA already reacts to other agents, and in the visible-robot setting to the robot. If randomizing its parameters covers the space of real reactions, our responder family adds nothing.
The strongest automated-curriculum competitor is DORAEMON run over those same ORCA and trajectory parameters. That comparison tells us whether gains come from the curriculum or from the responder family. The strongest interactive competitor is a learned-crowd scheme. Here pedestrians are reinforcement-learning agents updated alternately with the robot: a leader-follower scheme in the spirit of Oracles & Followers, plus a regret-driven adversary as in PAIRED. On the Labyrinth benchmark, PAIRED reached 40% success, against 0% for plain domain randomization. On crowd navigation, there is no reported result on Lacuna for either a learned-crowd or a PAIRED-style baseline.
All baselines receive the same robot architecture, the same number of robot environment steps, and the same hyperparameter search budget (equal numbers of seeds and configurations). Learned-crowd baselines are additionally reported at matched total wall-clock time, including pedestrian training, since compute is part of the claim.
Two ablations isolate the key ingredient. The first freezes the robot-avoidance and yield weights at zero, which keeps the same cost family but removes reactivity to the robot. The second replaces the inverse-Q-fitted prior with hand-set uniform ranges of equal volume.
Beating the field means two things. On the held-out evaluation described below, our method must have higher success and lower freezing rate than every baseline, with confidence intervals over at least five seeds that do not overlap. It must also use at most a small multiple of the default ORCA training cost, and clearly less than the learned-crowd baselines.
Why not just…?
Why not just uniformly randomize ORCA and social-force parameters? This is the real threat. Evidence from other domains does not settle it here. Uniform randomization failed in Domain Randomization via Entropy Maximization (Fixed-DR scored 0% on PandaPush) and in PAIRED (0% on Labyrinth). But those failures concern physics ranges and maze layouts, not crowd behavior. Our hypothesis is that ORCA's reactions are structurally narrow: every agent treats every other agent symmetrically and collision-free, so no setting of its parameters produces people who ignore the robot or who yield unevenly. The cross-family evaluation decides this directly. If ORCA-randomized training matches ours on held-out responder, social-force, and learned-pedestrian crowds, we drop the claim.
Why not just learn reactive pedestrians, adversarially or by alternating updates, on a fast GPU simulator? Learned crowds are more expressive, and parallel hardware makes them cheaper. However, Omni-Perception shows that even GPU simulators needed custom kernels to scale dynamic scenes. PAIRED's authors describe training three networks at once as computationally heavy. Adversarial pedestrians may also learn to corner the robot in ways real people do not. We report the learned-crowd baselines both at matched robot steps and at matched wall-clock time. If they win even at matched wall-clock time, the responder's advantage reduces to cost alone, and we will say so.
Why not just predict trajectories and plan around them? Intention Aware Robot Crowd Navigation already couples a trajectory predictor with a reinforcement-learning policy and transfers to a TurtleBot. Its predictor, though, is trained on simulated crowds whose reactions it cannot extend beyond. Purely reactive planners such as ORCA reach only 43% success in the setting above. We include the prediction-based policy as an additional robot architecture trained under each crowd condition. If our crowd family helps it too, the gain is in the training environment rather than the architecture.
Proposed Experiments
The decisive simulation test is a cross-family evaluation matrix. Every robot policy, trained under each crowd condition, is tested on five held-out crowd types:
- default ORCA
- social-force pedestrians
- responders whose parameters fall outside the training prior
- frozen learned pedestrian policies from the learned-crowd baseline
- non-reactive replayed trajectories
Each type is tested at 5, 10, and 20 humans, in both visible-robot and invisible-robot settings. The metrics are the CrowdNav ones: success rate, collision rate, extra navigation time, intrusion time ratio, and minimum separation distance. We add a freezing rate, defined as the fraction of episodes in which the robot's speed stays below 0.05 m/s for more than 3 s while its goal is not blocked.
The prediction is that our training wins mainly on the reactive held-out families and in dense crowds. That is where the default and ORCA-randomized policies should freeze or intrude. On non-reactive replay, we expect parity. If the gains are uniform across all families instead, they probably come from extra diversity rather than reactivity, and the zero-reactivity ablation will show which explanation holds.
Second, we measure cost directly: per-step simulation time, and total training wall-clock time to reach 90% success on the training crowd, for responder crowds versus learned crowds. This tests whether the inner loop has actually been removed. For reference, value-based inversion cut inverse reinforcement learning on Objectworld from over 8 hours to 1.77 minutes (Deep Inverse Q-learning with Constraints). We expect a comparable order-of-magnitude saving relative to retrained followers. If the saving is under a factor of three, the nested-optimization argument is weak.
Third, we report the prior fit itself. This includes the projection residual of the inverse-Q estimates and the coverage of recorded behaviors by $p_\phi$ before and after curriculum expansion.
Finally, we deploy zero-shot on a mobile base comparable to the TurtleBot 2i used in Intention Aware Robot Crowd Navigation, which reports no real-world numbers on Lacuna. We compare the default-ORCA policy, the ORCA-randomized policy, and ours in corridor and open-hall crossings with naive volunteers. We measure success, minimum separation, intrusion time, and freezing rate over at least 30 trials per condition, with condition order randomized. The hardware result counts only if it agrees with the direction of the simulation matrix. A method that wins on hardware but not on held-out simulated crowds would point to a confound, not to a better crowd model.
References
- Deep Inverse Q-learning with Constraints
- Relational Graph Learning for Crowd Navigation
- Omni-Perception: Omnidirectional Collision Avoidance of Legged Robots in Dynamic Environments
- Oracles & Followers: Stackelberg Equilibria in Deep Multi-Agent Reinforcement Learning
- Inverse Optimal Control with Linearly-Solvable MDPs
- IQ-Learn: Inverse soft-Q Learning for Imitation
- Domain Randomization via Entropy Maximization
- Crowd-Robot Interaction: Crowd-Aware Robot Navigation With Attention-Based Deep Reinforcement Learning
- Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design
- Intention Aware Robot Crowd Navigation with Attention-Based Interaction Graph