Abstract
Effective leveraging of real-world driving datasets is crucial for enhancing the training of autonomous driving systems. While Offline Reinforcement Learning enables the training of autonomous vehicles using such data, most available datasets lack meaningful reward labels. Reward labeling is essential as it provides feedback for the learning algorithm to distinguish between desirable and undesirable behaviors, thereby improving policy performance. This paper presents a novel pipeline for generating human-aligned reward labels. The proposed approach addresses the challenge of absent reward signals in real-world datasets by generating labels that reflect human judgment and safety considerations. The pipeline incorporates an adaptive safety component, activated by analyzing semantic segmentation maps, allowing the autonomous vehicle to prioritize safety over efficiency in potential collision scenarios. The proposed pipeline is applied to an occluded pedestrian crossing scenario with varying levels of pedestrian traffic, using synthetic and simulation data. The results indicate that the generated reward labels closely match the simulation reward labels. When used to train the driving policy using Behavior Proximal Policy Optimisation, the results are competitive with other baselines. This demonstrates the effectiveness of our method in producing reliable and human-aligned reward signals, facilitating the training of autonomous driving systems through Reinforcement Learning outside of simulation environments and in alignment with human values.
Introduction
Reinforcement Learning (RL) has emerged as a powerful framework for teaching autonomous vehicles (AVs) complex decision-making skills. However, traditional "online" RL requires agents to learn through trial and error within a simulator, which often fails to capture the nuances of real-world environments—a problem known as the "sim-to-real gap." While the industry possesses vast "offline" datasets of real-world driving logs (e.g., Waymo, nuScenes), these datasets lack the reward labels necessary for RL training.
This paper introduces a novel pipeline designed to generate human-aligned reward labels from these static logs. By focusing on the "occluded pedestrian crossing" scenario—one of the most dangerous challenges in urban driving—the researchers demonstrate how automated perception can create high-quality training signals that align with human safety values.
Methodology
The proposed framework consists of two primary innovations: a Reward Labeling Pipeline and a Spatial Attention Mechanism.
1. Human-Aligned Reward Labeling
To move beyond simple binary rewards, the system utilizes a ResNet18-based UNet to generate semantic segmentation maps from camera data. These maps are used to calculate an Adaptive Safety Component ($c_t$) based on three critical risk factors:
- Pedestrian Presence: Detection of individuals on the road or sidewalk.
- Zebra Crossing Presence: Identification of designated crossing areas, even when partially obscured.
- Pedestrian History: A temporal memory bank that tracks pedestrians who may have recently disappeared behind obstacles, such as parked vans.
If the calculated risk probability exceeds a predefined threshold, the reward function dynamically shifts its priority from efficiency (maintaining target speed) to safety (prioritizing braking).
2. Spatial Attention Mechanism
To ensure the RL agent focuses on the most relevant environmental features, the authors implemented a spatial attention weighting system based on the semantic maps. Importance weights are assigned as follows:
- Pedestrians: 1.0 (Highest priority)
- Zebra Crossings: 0.75
- Other Vehicles: 0.5
This weighting guides the agent to prioritize salient features necessary for effective emergency braking during the training process.
3. Offline RL Training
The generated labels were used to train three prominent Offline RL algorithms: Conservative Q-Learning (CQL), Implicit Q-Learning (IQL), and Behavioral Proximal Policy Optimization (BPPO).
Experimental Evaluation
The pipeline was tested using a combination of synthetic and real-world data:
- CARLA Simulator: 20,000 images were generated to stress-test the system across low, medium, and high pedestrian traffic densities.
- A2D2 (Audi Autonomous Driving Dataset): 1,315 real-world samples involving pedestrian crossings were used to validate the labeling method against human judgment.
Key Results and Insights
The "Machine Eye" Advantage
A significant finding of the study was that the automated labeling system frequently outperformed human annotators. In several instances, the pipeline flagged unsafe states involving heavily occluded pedestrians that the human reviewer missed. This suggests that perception-based automated labeling can provide more consistent and vigilant safety signals than manual review.
Algorithm Performance
Among the tested algorithms, Implicit Q-Learning (IQL) emerged as the most robust. It maintained a 91-100% success rate across various traffic densities and occlusion levels. Policies trained with the proposed "Gen" labeling method maintained safe stopping distances (averaging 3.8m to 4.4m), avoiding the overly aggressive behaviors seen in baseline models.
Comparison with Vision Language Models (VLMs)
The researchers compared their pipeline against a VLM baseline. They found that while VLMs are capable of reasoning, they act as "black boxes" that are 120x slower to process. Furthermore, VLMs struggled to produce the smooth, continuous reward signals required for stable RL training, making the proposed segmentation-based approach superior for this application.
Conclusion
This research demonstrates that human-aligned reward labeling can effectively transform offline driving data into a viable training resource for RL. By incorporating temporal context (pedestrian history) and spatial attention, the system produces agents that prioritize safety in high-risk, occluded scenarios. Future work will focus on expanding the system's situational awareness by integrating 360-degree multi-camera feeds and utilizing Video Transformer architectures to better predict the trajectories of other road users.
Summary updated