Dynamic-Weighted Simplex Strategy for Learning Enabled Cyber Physical\n Systems

Shreyas Ramakrishna, Charles Hartsell, Matthew P Burruss, Gábor Karsai, Abhishek Dubey

TL;DR

This paper introduces a "dynamic-weighted simplex strategy" that uses reinforcement learning to smoothly blend the outputs of a high-performance learning-based controller and a safe, traditional controller for autonomous systems. This approach significantly reduces safety violations (out-of-track occurrences) while maintaining higher speeds compared to using either controller alone, demonstrated on a custom-built RC car testbed.

What Problem Does This Paper Address?

Autonomous systems, particularly Cyber-Physical Systems (CPS) like self-driving cars, increasingly rely on Learning Enabled Components (LECs) for tasks like perception and control. While LECs offer impressive capabilities and continuous learning, they are notoriously difficult to assure for safety. The traditional "Simplex Architecture" addresses this by combining a high-performance (but untrusted) advanced controller (AC) with a safe baseline controller (BC) and a decision manager (DM) that switches between them. However, this architecture faces two key challenges: designing an effective decision logic for the DM, and the abrupt, jarring transitions between controllers that can degrade system performance and passenger comfort. This paper aims to overcome these limitations by creating a smoother, more adaptive way to combine controller outputs.

Methods

The core innovation is the dynamic-weighted simplex strategy. Instead of a hard switch, this strategy uses a weighted ensemble of controller outputs, where the weights can vary dynamically. This allows for smoother transitions between the advanced controller (an end-to-end learned CNN for steering) and the baseline controller (an OpenCV-based lane detection system).

To determine these dynamic weights, the authors employ reinforcement learning (RL), specifically Q-learning. The RL agent learns an optimal policy to adjust the weights (WL for the LEC, WC for the OpenCV controller) and the vehicle's speed based on the current state (weights, speed, steering commands). The reward function is designed to balance performance (vehicle speed) with safety (keeping the car within track boundaries).

A middleware framework is developed to deploy this strategy on a physical platform. This framework includes a resource manager that monitors the system's computational resources (CPU, temperature) and can offload tasks to external "fog" devices to prevent performance degradation due to overheating or overload.

Datasets

The paper uses data collected from the DeepNNCar testbed. Training data for the LEC controller (a modified NVIDIA DAVE-II CNN) was collected from two indoor tracks (Track 1 and Track 2), totaling 6000 labeled images. Validation was performed on a separate dataset of 1000 images. The dynamic-weighted simplex strategy was trained using RL on Track 1, and then tested on Track 3 to evaluate generalization.

Key Insights and Ideas

  • Smooth transitions are crucial for performance: Abrupt switching between controllers in traditional Simplex Architectures can lead to jerky movements and reduced performance. The weighted ensemble approach offers a more graceful way to blend controller outputs.
  • RL is a powerful tool for adaptive control: By framing the problem as an RL task, the system can learn to dynamically adjust controller weights and speed based on context (e.g., track segment), leading to better overall performance and safety.
  • Resource management is essential for embedded CPS: The computational demands of advanced control strategies can overwhelm embedded platforms. A proactive resource manager that monitors and offloads tasks is vital for maintaining performance and preventing hardware issues like overheating.
  • Context matters for optimal control: The learned weights for the dynamic-weighted simplex strategy vary significantly across different track segments (straight vs. curved), highlighting the importance of context-aware control.

Key Results

  • The dynamic-weighted simplex strategy achieved 60% fewer out-of-track occurrences compared to the original LEC-driven system.
  • It demonstrated higher optimized speed (0.4 m/s) during indoor driving than the original LEC system.
  • The dynamic-weighted simplex strategy consistently outperformed individual controllers (LEC and OpenCV) and the fixed-weighted simplex strategy in terms of reducing out-of-track occurrences, especially at higher speeds.
  • The resource manager effectively mitigated RPi3 overheating by offloading tasks, though this led to a slight decrease in maximum speed (0.34 m/s with offloading vs. 0.42 m/s onboard).
  • Inference times for the weighted simplex strategies were higher (around 120-130 ms) than for individual controllers (around 80 ms) due to the dual controller operation and decision computation.

Limitations and Future Work

  • Soft constraints: The current approach is designed for systems that can tolerate "soft" constraint violations (where violations incur a penalty). Extending this to "hard" constraints (where violations are unacceptable) is a future direction.
  • Offline learning: The dynamic weights are currently learned offline. Developing an online learning technique to adapt weights on the fly would allow the system to handle new, unseen contexts (e.g., varying lighting conditions).
  • Network failure: The middleware framework's reliance on wireless communication makes it vulnerable to network failures. The authors suggest exploring architectures like CHARIOT for autonomous resilience.
  • State space exploration: For systems with larger state spaces, more extensive exploration phases might be needed to ensure robust Q-tables.
  • Track complexity: The current test tracks were small, limiting the exploration of all possible steering combinations. Larger, more complex tracks would provide a more comprehensive evaluation.

Summary updated

Abstract

Cyber Physical Systems (CPS) have increasingly started using Learning Enabled\nComponents (LECs) for performing perception-based control tasks. The simple\ndesign approach, and their capability to continuously learn has led to their\nwidespread use in different autonomous applications. Despite their simplicity\nand impressive capabilities, these models are difficult to assure, which makes\ntheir use challenging. The problem of assuring CPS with untrusted controllers\nhas been achieved using the Simplex Architecture. This architecture integrates\nthe system to be assured with a safe controller and provides a decision logic\nto switch between the decisions of these controllers. However, the key\nchallenges in using the Simplex Architecture are: (1) designing an effective\ndecision logic, and (2) sudden transitions between controller decisions lead to\ninconsistent system performance. To address these research challenges, we make\nthree key contributions: (1) \\textit{dynamic-weighted simplex strategy} -- we\nintroduce ``weighted simplex strategy" as the weighted ensemble extension of\nthe classical Simplex Architecture. We then provide a reinforcement learning\nbased mechanism to find dynamic ensemble weights, (2) \\textit{middleware\nframework} -- we design a framework that allows the use of the dynamic-weighted\nsimplex strategy, and provides a resource manager to monitor the computational\nresources, and (3) \\textit{hardware testbed} -- we design a remote-controlled\ncar testbed called DeepNNCar to test and demonstrate the aforementioned key\nconcepts. Using the hardware, we show that the dynamic-weighted simplex\nstrategy has 60\\% fewer out-of-track occurrences (soft constraint violations),\nwhile demonstrating higher optimized speed (performance) of 0.4 m/s during\nindoor driving than the original LEC driven system.\n
BibTeX
@inproceedings{ramakrishna2019dynamic,
  author = {Shreyas Ramakrishna and Charles Hartsell and Matthew P Burruss and Gábor Karsai and Abhishek Dubey},
  title = {{Dynamic-Weighted Simplex Strategy for Learning Enabled Cyber Physical\textbackslash\{\}n Systems}},
  booktitle = {ISORC},
  year = {2019},
  doi = {10.1109/isorc.2019.00032},
}
Figure