Learning to park from logged driving is attractive because logs are cheap. The logs are also messy. Drivers make extra back-and-forth shuffles, hesitate, and sometimes clip an obstacle. Methods that learn from such imperfect demonstrations score each logged action by how much better it looks than the alternatives, then copy good actions more and bad ones less. For a car, most "alternatives" are physically impossible. A car cannot slide sideways, and in a tight spot only a handful of steering-and-gear choices avoid a collision. We propose to score every demonstrated action only against the kinematically feasible actions in that state, and to set the policy's randomness by how many feasible actions exist there. A forced move in a bottleneck is then never mistaken for a bad choice. Real choices in open space still reveal how skilled the driver was.
This matters because the two tools already exist separately and fail in complementary ways. Suboptimal-demonstration learners such as Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations and Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching were tested only on locomotion and hand manipulation and know nothing about steering geometry. The parking planner HOPE prunes infeasible steering with a geometric action mask but trains purely online and cannot use logs. Our claim is that the mask is more than a deployment-time safety filter. It is also the correct reference set for judging demonstrator quality and the correct scale for policy entropy.
Core Claim
Three terms need defining first. Non-holonomic constraints are velocity constraints, such as "wheels roll but do not slip sideways," that cannot be reduced to constraints on position alone. They are why parallel parking needs multi-point maneuvers. Maximum entropy reinforcement learning adds a bonus for policy randomness to the reward, so the optimal policy becomes a Boltzmann distribution over action values instead of a single action. The expert optimality assumption is the premise in imitation learning that every demonstrated action is the one to copy.
Take a discretized action space of steering angle, step length and gear, as in HOPE. Let $M(s)$ be the set of actions in state $s$ that respect the steering limits and keep collision clearance, computed exactly from vehicle geometry. Let $Q(s,a)$ be a learned action value and $\alpha$ a temperature. We define the masked soft value and masked advantage:
$$V_M(s) = \alpha \log \sum_{a' \in M(s)} \exp\big(Q(s,a')/\alpha\big), \qquad A_M(s,a) = Q(s,a) - V_M(s).$$
Demonstrated pairs are weighted by $\exp(\beta A_M(s,a))$, as in the exponentially weighted imitation of MARWIL. The only change is that the baseline sums over feasible actions rather than all actions. During online fine-tuning, the entropy target in state $s$ is set to $\kappa \log |M(s)|$, a fixed fraction $\kappa$ of the largest entropy possible over the feasible set, instead of a single constant.
The mechanism we expect to matter is as follows. Offline value functions overestimate actions the data never contains, which D4RL documents as the reason offline SAC fails. In a bottleneck, almost every unseen action is infeasible. An unmasked soft value is therefore inflated precisely there. That pushes the advantage of the demonstrator's forced, correct maneuver negative, and the learner downweights the most important states in the dataset.
Masking removes this bias using known physics rather than a learned support penalty. It also keeps feasible-but-unseen actions in the baseline, so the policy can still improve beyond the demonstrator. Setting entropy by $\log|M(s)|$ gives a geometry-derived version of the state-dependent rationality weight $\mu(s)$ that Weighted Maximum Entropy Inverse Reinforcement Learning learns from data. That method was evaluated on taxi road graphs and MuJoCo, never on continuous vehicle kinematics.
Research Program
Relation to prior work. The work builds on four method sources. MARWIL supplies the advantage-weighted cloning loss. HOPE supplies the action mask, which we reimplement on its Tactics2D parking environment. Soft Q-learning from Reinforcement Learning with Deep Energy-Based Policies supplies the Boltzmann policy form. Because the action set is discrete and masked, the partition function is an exact finite sum. This avoids the sampling machinery and instability that paper reports for continuous energy-based policies.
The closest system is DRIP. It pretrains SAC or PPO, keeps only successful rollouts, and refines them with a diffusion model. It handles demonstrator quality by binary rejection and never uses kinematics when deciding which data to trust. We differ in three ways. We use all logged data, with feasibility-aware soft weights. We discard nothing; collisions still train $Q$. We never denoise in an unconstrained action space.
TRAIL motivates keeping failed data. It shows that even the bottom 5% of offline data teaches useful dynamics. We take a narrower and cheaper route: the dynamics of a car are already known, and the scarce information is which feasible choices are good.
Data. There is no public log of imperfect human parking on the HOPE benchmark, so the first deliverable is a graded suboptimal dataset built on the Dragon Lake Parking scenarios used by HOPE and DRIP. It mixes several sources:
- rollouts from HOPE checkpoints at several training stages;
- Hybrid A* with injected steering jitter;
- Reeds-Shepp maneuvers with random extra direction switches, mimicking a hesitant driver;
- a share of colliding rollouts.
These are combined at four quality levels, following the L1–L4 protocol of RelaxDICE. Because the generator is known, we can label each transition as forced (|M(s)| small) or discretionary. We can also check whether the learned weights track true demonstrator quality.
Training. The method has two stages. Offline, $Q$ is fit with a masked soft Bellman backup, and the policy is fit by masked advantage-weighted cloning. Online, the policy is fine-tuned with masked SAC using the feasibility-scaled entropy target. The offline-to-online step matters because the file reports sharp entropy drops when moving from imitation to RL (Xiong et al.). Parking also has real multimodality, such as head-in versus reverse-in, which collapsed entropy would erase.
Analysis. We will measure how advantage weights distribute across states binned by $|M(s)|$, under masked and unmasked baselines. If the claimed bias exists, unmasked weights will be systematically low in small-$|M(s)|$ states, even for transitions generated by the best demonstrator.
Baselines
Naive reference. Vanilla behavioral cloning on the full mixed dataset, with HOPE's mask applied at execution. It shows how much any weighting buys. MARWIL's own TORCS result (score 2710 for plain cloning versus 9492 with weighting, from Exponentially Weighted Imitation Learning for Batched Historical Data) suggests the gap can be large. Parking may differ.
Strongest suboptimal-imitation competitors. DWBC and RelaxDICE, run on the identical dataset with test-time masking. DWBC beat ORIL, BCND and DemoDICE in 27 of 36 D4RL tasks. RelaxDICE-DRC won 18 of 22 settings and beat DemoDICE by over 30% at the hardest suboptimality levels. These are the strongest published methods for exactly the expert-optimality question, so they are the main comparison.
Unmasked ablation. MARWIL with an unmasked soft value, with the mask applied only at test time. This ablation isolates the key ingredient: whether masking the baseline, rather than masking execution, is what helps.
Learned-entropy ablation. Our method with the entropy target replaced either by a constant or by a learned $\mu(s)$ in the style of W-MaxEnt. This separates the value of the geometry-derived entropy schedule.
Strongest planning systems. HOPE and DRIP as published references. HOPE reports over 97% overall success and 94.2% on extreme parallel parking, against 16.8% for Hybrid A*. DRIP reports 85.9% overall success on 7,000 DLP scenes, against 75.5% for SAC and 75.9% for Diffusion Q-Learning, and 65.9% in hard confined scenes.
HOPE uses unlimited online simulation, so it is not an apples-to-apples competitor in the offline phase. We will compare against it after our online fine-tuning, matching environment interaction steps. We will also report how many online steps each method needs to reach a given success rate.
Fairness. All learned methods share the same network, the same data, the same mask, and an equal hyperparameter search budget over $\beta$, $\alpha$ and the method-specific thresholds.
What counts as beating them. Higher success on the Complex and Extreme tiers at L3 and L4 data quality than DWBC and RelaxDICE. Beyond that, reaching HOPE's reported success level with a stated fraction of HOPE's online interaction.
Why not just…?
Why not just apply HOPE's mask at test time on top of DWBC or RelaxDICE? This is the most direct threat, and it is run as a baseline. Test-time masking prevents infeasible actions but cannot repair weights that were computed against impossible alternatives. If those methods have already learned to distrust bottleneck maneuvers, masking leaves only the feasible actions they rank badly. If test-time masking matches our method, the claimed bias is absent in practice, and we would report that the mask's value lies only in execution.
Why not just use a pessimistic offline RL method such as CQL, which D4RL found strongest on AntMaze and Adroit? CQL suppresses unseen actions with a learned penalty. In a car, the set of truly impossible actions is known exactly, so learning it wastes data. A penalty on all unseen actions also suppresses feasible maneuvers the demonstrators never tried, which limits improvement over them. This is a real competitor, so we run masked CQL as an additional baseline. If it equals our method, the contribution shrinks to "masks help any offline learner," which is still useful but smaller.
Why not just filter to successful rollouts as DRIP does? At L3 and L4 few rollouts succeed cleanly. Many failed rollouts also contain correct forced maneuvers followed by a late mistake. CAIL (Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality) shows that piecing together partial quality beats discarding. Our success-filtered-BC baseline at each quality level will show whether filtering degrades as data worsens, which is our prediction.
Why not just run SAC from scratch with the mask? That is essentially HOPE's regime and works with enough simulation. The question here is how far logs cut interaction. The interaction-to-success curves answer it directly.
Proposed Experiments
Main experiment. Success rate, collision rate, gear-switch count and path length on the HOPE/DLP difficulty tiers, for every method, across the four data quality levels, with five seeds each.
Mechanism check. Plot the mean advantage weight against $|M(s)|$ for transitions from the best generator only. Our account predicts unmasked weights fall in small-$|M(s)|$ states while masked weights stay flat. If both are flat, the core claim is wrong.
Entropy check. Track policy entropy and the coverage of head-in versus reverse-in solutions during fine-tuning, comparing constant, learned and feasibility-scaled targets. Success in the Extreme tier should rise without entropy collapse in open bays.
Transfer check. Repeat the offline comparison on the CARLA E2E Parking benchmark, with a kinematic mask derived from the vehicle model. That paper reports 85.16% target success and a 2.86% collision rate for its model, against roughly 75% and 9.37% for its baseline. We would treat these as reference points rather than direct competitors, since our inputs are not end-to-end camera images.
References
- Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
- Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
- HOPE: A Reinforcement Learning-Based Hybrid Policy Path Planner for Diverse Parking Scenarios
- Exponentially Weighted Imitation Learning for Batched Historical Data
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning.
- Weighted Maximum Entropy Inverse Reinforcement Learning
- Reinforcement Learning with Deep Energy-Based Policies
- A Diffusion-Refined Planner with Reinforcement Learning Priors for Confined-Space Parking
- TRAIL: Near-Optimal Imitation Learning with Suboptimal Data
- Scheduled Policy Optimization for Natural Language Communication with Intelligent Agents
- Confidence-Aware Imitation Learning from Demonstrations with Varying Optimality
- E2E Parking Dataset: An Open Benchmark for End-to-End Autonomous Parking