TL;DR
This paper introduces a "dynamic-weighted simplex strategy" that uses reinforcement learning to smoothly blend the outputs of a high-performance learning-based controller and a safe, traditional controller for autonomous systems. This approach significantly reduces safety violations (out-of-track occurrences) while maintaining higher speeds compared to using either controller alone, demonstrated on a custom-built RC car testbed.
What Problem Does This Paper Address?
Autonomous systems, particularly Cyber-Physical Systems (CPS) like self-driving cars, increasingly rely on Learning Enabled Components (LECs) for tasks like perception and control. While LECs offer impressive capabilities and continuous learning, they are notoriously difficult to assure for safety. The traditional "Simplex Architecture" addresses this by combining a high-performance (but untrusted) advanced controller (AC) with a safe baseline controller (BC) and a decision manager (DM) that switches between them. However, this architecture faces two key challenges: designing an effective decision logic for the DM, and the abrupt, jarring transitions between controllers that can degrade system performance and passenger comfort. This paper aims to overcome these limitations by creating a smoother, more adaptive way to combine controller outputs.
Methods
The core innovation is the dynamic-weighted simplex strategy. Instead of a hard switch, this strategy uses a weighted ensemble of controller outputs, where the weights can vary dynamically. This allows for smoother transitions between the advanced controller (an end-to-end learned CNN for steering) and the baseline controller (an OpenCV-based lane detection system).
To determine these dynamic weights, the authors employ reinforcement learning (RL), specifically Q-learning. The RL agent learns an optimal policy to adjust the weights (WL for the LEC, WC for the OpenCV controller) and the vehicle's speed based on the current state (weights, speed, steering commands). The reward function is designed to balance performance (vehicle speed) with safety (keeping the car within track boundaries).
A middleware framework is developed to deploy this strategy on a physical platform. This framework includes a resource manager that monitors the system's computational resources (CPU, temperature) and can offload tasks to external "fog" devices to prevent performance degradation due to overheating or overload.
Datasets
The paper uses data collected from the DeepNNCar testbed. Training data for the LEC controller (a modified NVIDIA DAVE-II CNN) was collected from two indoor tracks (Track 1 and Track 2), totaling 6000 labeled images. Validation was performed on a separate dataset of 1000 images. The dynamic-weighted simplex strategy was trained using RL on Track 1, and then tested on Track 3 to evaluate generalization.
Key Insights and Ideas
- Smooth transitions are crucial for performance: Abrupt switching between controllers in traditional Simplex Architectures can lead to jerky movements and reduced performance. The weighted ensemble approach offers a more graceful way to blend controller outputs.
- RL is a powerful tool for adaptive control: By framing the problem as an RL task, the system can learn to dynamically adjust controller weights and speed based on context (e.g., track segment), leading to better overall performance and safety.
- Resource management is essential for embedded CPS: The computational demands of advanced control strategies can overwhelm embedded platforms. A proactive resource manager that monitors and offloads tasks is vital for maintaining performance and preventing hardware issues like overheating.
- Context matters for optimal control: The learned weights for the dynamic-weighted simplex strategy vary significantly across different track segments (straight vs. curved), highlighting the importance of context-aware control.
Key Results
- The dynamic-weighted simplex strategy achieved 60% fewer out-of-track occurrences compared to the original LEC-driven system.
- It demonstrated higher optimized speed (0.4 m/s) during indoor driving than the original LEC system.
- The dynamic-weighted simplex strategy consistently outperformed individual controllers (LEC and OpenCV) and the fixed-weighted simplex strategy in terms of reducing out-of-track occurrences, especially at higher speeds.
- The resource manager effectively mitigated RPi3 overheating by offloading tasks, though this led to a slight decrease in maximum speed (0.34 m/s with offloading vs. 0.42 m/s onboard).
- Inference times for the weighted simplex strategies were higher (around 120-130 ms) than for individual controllers (around 80 ms) due to the dual controller operation and decision computation.
Limitations and Future Work
- Soft constraints: The current approach is designed for systems that can tolerate "soft" constraint violations (where violations incur a penalty). Extending this to "hard" constraints (where violations are unacceptable) is a future direction.
- Offline learning: The dynamic weights are currently learned offline. Developing an online learning technique to adapt weights on the fly would allow the system to handle new, unseen contexts (e.g., varying lighting conditions).
- Network failure: The middleware framework's reliance on wireless communication makes it vulnerable to network failures. The authors suggest exploring architectures like CHARIOT for autonomous resilience.
- State space exploration: For systems with larger state spaces, more extensive exploration phases might be needed to ensure robust Q-tables.
- Track complexity: The current test tracks were small, limiting the exploration of all possible steering combinations. Larger, more complex tracks would provide a more comprehensive evaluation.
Summary updated