Lane-Branch Actions as a Sim-to-Real Bridge for Mapless Driving

Self-driving systems that cannot rely on expensive pre-surveyed maps have to do two things at once. They must build the local road layout from their own cameras and LiDAR (a laser range sensor), and they must decide how to drive on it. Training that decision-making in a simulator is attractive because simulators supply unlimited, safe, fully labelled experience. The catch is the "reality gap": policies learn simulator quirks and then fail on real sensor data. The usual remedy is domain randomization, which means scrambling lighting, textures, noise and physics during training so the policy cannot latch onto any one rendering. There is direct evidence that this remedy is weak for driving sensors. S2R-Bench compared real camera, LiDAR and radar degradations with simulated ones and found that simulated glare badly understates the real LiDAR failure. Simulated sensor misalignment also fails to match the real performance drop of a fusion detector.

This note proposes a different lever. It changes what the policy is allowed to decide rather than making the rendered inputs more varied. The driving policy makes a discrete choice of which lane centerline to follow. The candidates come from a lane graph the car builds online, anchored to a cheap road-level navigation map such as OpenStreetMap. Given that choice, the policy outputs a smooth, continuous trajectory along the chosen lane. The discrete choice means the same thing in simulation and in reality, even when the pixels do not. During simulator training only, a critic with access to the true high-definition (HD) map, the precise, centimeter-level lane layout, scores these choices. At deployment the car uses only its sensors and the coarse map. The hypothesis is that this lane-branch action space shrinks the sim-to-real gap of a learned planner more than extra sensor randomization does. It should do so while avoiding the jerky motion that discretized controls normally cause.

Core Claim

The claim has three parts. First, when a planner trained only in simulation is evaluated on real driving logs, the drop relative to the same planner trained on real logs is substantially smaller if its actions are "pick a lane branch, then emit a continuous trajectory in that lane's frame" than if its actions are free-form waypoints. Second, this holds with identical sensors, randomization and training budget. Third, the smoothness penalty that makes discrete driving controls uncomfortable disappears when the continuous part is a continuous-time function rather than a waypoint chunk.

The analogy that motivates the claim comes from robot assembly. Learning Insertion Primitives pairs a discrete choice of primitive (translate, rotate, insert) with continuous parameters, and ends each primitive on a force threshold rather than a timer. That design transferred zero-shot from MuJoCo to a real industrial arm, including to connectors it never trained on. The part of the action that is defined by a physical event transferred. We argue that the lane-graph branch plays the same role in driving: it is an abstraction whose meaning does not depend on the renderer.

Research Program

The system has three pieces. The perception front end fuses multi-camera images with LiDAR into a set of directed lane centerlines with lane-to-lane connectivity. Each centerline is represented as a Bezier curve, following TopoMLP, which showed that detection quality, not elaborate graph reasoning, caps topology performance on OpenLane-V2. Those centerlines cross-attend to a road-level map skeleton, as in P-MapNet and the association approach of Online Navigation Refinement. P-MapNet fused OpenStreetMap priors into camera-based map generation for long range. Online Navigation Refinement matched road-level navigation paths to perceived lanes with a transformer and topology-aware beam search. We reuse both ideas as components. Neither paper feeds the result to a learned controller, and neither trains in simulation.

The action head scores each predicted centerline as a discrete branch, so the size of the action set grows with the local map rather than being fixed. Two extra branches are always available. One follows the coarse map skeleton when no perceived lane matches. The other is a free-space branch that follows no lane. These exist because perception will sometimes miss the correct lane, and without them the policy could not recover.

Conditioned on the chosen branch, a small sinusoidal implicit network outputs lateral offset and speed as continuous functions of time in that lane's coordinate frame. An implicit network here means one that represents a trajectory as a function you can query at any time. This part reuses the method of Neural Implicit Action Fields. That work replaced waypoint chunks with continuous functions for tabletop manipulation. Because the functions have analytic derivatives, it could supervise velocity and jerk directly and remove stop-and-go jitter. We apply it to vehicle motion inside a topological choice, which that work does not consider.

Training happens in the CARLA driving simulator with an asymmetric actor-critic, following Asymmetric Actor Critic and NavDP. The first gave the critic full simulator state while the actor saw only images, and transferred zero-shot to a real arm. The second used privileged distance fields and planners to guide a diffusion navigation policy across real robots, but had no road map and noted that its lack of global memory limits long-range navigation. Our critic sees CARLA's ground-truth lane graph and all agent states. It estimates the value of each branch together with its continuous parameters, in the style of the parameterized Q-learning used for insertion primitives. The actor sees only sensors and a road-level map. We make that road-level map by coarsening CARLA's lane graph to one line per road, which mirrors the information in OpenStreetMap.

Domain randomization covers four things: appearance and weather, LiDAR beam dropout and intensity, camera-to-LiDAR extrinsic jitter, and corruption of the coarse map (lateral offsets, missing roads, wrong lane counts). Randomizing the coarse map is the least-studied of these. It targets a known failure: cross-attention to a misaligned prior can hallucinate roads.

Baselines

The naive reference point is a modular, non-learned stack. It uses the same fused online lane graph, picks the branch by Online Navigation Refinement's beam-search association to the route, and follows that branch with a fixed spline-and-speed controller. It needs no simulator training at all, so it sets the floor that any learned planner must clear. For branch selection, the file reports that the Map Association Transformer reaches an NR-F1 of 78.7 on the OMA validation set, against 70.1 for a hidden-Markov-model matcher (Online Navigation Refinement). Our branch selection must at least match 78.7 on OMA to claim it is not degrading the association step.

The strongest existing learned planner is VAD. It predicts vectorized map elements and agent motion, then plans a continuous trajectory under explicit collision and lane-boundary constraints. It is the closest system that couples online vector maps with planning. It was trained on real nuScenes logs and tested closed-loop in CARLA as a separate step. The file reports that VAD-Base reduced the average open-loop collision rate by 29.0% relative to UniAD on nuScenes. It also reports that VAD had the highest CARLA Town05 Driving Score among vision-only methods, but gives no absolute value, so no reported result was found on Lacuna for VAD's absolute nuScenes collision rate or CARLA Driving Score.

We will retrain VAD three ways. The first uses nuScenes, which is its reference setting. The second uses our CARLA data with our randomization. The third feeds VAD the same camera-plus-LiDAR features as our model, so modality differences do not confound the comparison. All three share our backbone size, GPU budget and a matched hyperparameter search. Beating VAD means the following. A version of our planner trained on CARLA alone must come within a small margin of VAD trained on nuScenes on nuScenes open-loop collision rate and L2 error. It must also clearly outperform VAD trained on CARLA.

The ablation that isolates the key ingredient removes the discrete branch while keeping everything else fixed: the same perception, critic, randomization and implicit trajectory head. The actor then emits one continuous trajectory in the ego frame. A second ablation replaces branches with fixed buckets of steering angle and acceleration, to separate "discrete" from "discrete over map topology." A third makes the critic symmetric, with no ground-truth HD map, to measure what the privileged map contributes. A fourth swaps the implicit trajectory function for waypoint chunks, to measure jerk.

Why not just…?

Why not just train a continuous planner in simulation with heavier randomization? This is the no-branch ablation above, and it is the main threat. S2R-Bench suggests that sensor randomization cannot cover real glare and misalignment, because the simulated corruptions miss real failures by large margins. More randomization therefore will not by itself close the gap. We still test this directly by doubling the randomization strength and range for the continuous baseline. If that version closes the transfer gap as well as the branch action space does, our central claim fails.

Why not just use the modular stack: online map, route association, then a hand-tuned lane follower? This is a real competitor on benign roads, and it is our naive baseline. We expect it to fail where the decision depends on other agents. Examples are merges, yielding at unprotected turns, and choosing the free-space branch around a blocked lane, because a fixed follower has no model of interaction. CARLA closed-loop infraction counts on scenarios with dense traffic will settle the question. If the modular stack matches our planner there, the learned component is not worth its cost and we will report that.

Why not just train on real fleet logs, as VAD does? Real logs remove the reality gap for the conditions they contain. Commercial fleet data, however, is noisy, skewed toward benign driving, and needs heavy cleaning, as documented by CleanMAP and the Seoul bus and taxi pipeline. Logs also cannot supply ground-truth lane graphs for a privileged critic, or closed-loop trial and error. We do not claim simulation beats real data outright. We claim that the branch action space makes simulation experience transfer. We will also test simulation pre-training followed by real fine-tuning on nuScenes against real-only training, at equal real-data budgets.

Why not just discretize steering and throttle, which makes tree search and safety shields easy? Fixed control buckets are known to produce unsmooth velocity profiles. The bucket ablation, scored on jerk and on collision rate, will show whether topology-level discreteness keeps the benefits of discreteness without that comfort cost.

Proposed Experiments

Training uses CARLA towns with camera, LiDAR and randomized weather. Towns and weather presets are held out for simulator testing. The zero-shot real test is open-loop on nuScenes, reporting L2 trajectory error, collision rate and a jerk statistic computed from the trajectory function's analytic derivatives. Branch quality on real data is scored on OMA (NR-F1) and lane-graph quality on OpenLane-V2 subset_B, which is built on nuScenes. For context, TopoMLP reports 41.2% OLS, but on subset_A with a ResNet-50 backbone. The file gives no subset_B figure for TopoMLP, so no reported result was found on Lacuna for OpenLane-V2 subset_B, and we will measure it ourselves.

The decisive quantity is the transfer gap: nuScenes performance of a CARLA-trained planner minus that of the identical planner trained on nuScenes. It is computed for our model and for each ablation. The hypothesis succeeds if the branch action space cuts this gap by a clear, seed-robust margin relative to the no-branch ablation under matched randomization. It fails if the gaps are statistically indistinguishable.

Two further checks probe mechanism. The first bins nuScenes frames by whether the ground-truth lane was present in the predicted candidate set. Given TopoMLP's finding that detection is the bottleneck, we expect most residual errors to fall in the "missing lane" bin, and the skeleton and free-space branches should recover part of that loss. The second drops LiDAR or cameras at test time and corrupts the road-level map with real OMA misalignments. This measures whether randomizing the coarse map during training is what prevents hallucinated branches. Results are only as strong as nuScenes open-loop evaluation allows, and no physical vehicle is used. We state that limitation plainly rather than calling open-loop log replay closed-loop real deployment.