Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers

Aryaman Gupta, Kaustav Chakraborty, Somil Bansal

Abstract

Vision-based autonomous controllers often struggle with out-of-distribution (OOD) inputs, leading to potential safety violations. This paper proposes a novel framework that shifts focus from component-level errors to system-level anomalies. By integrating Hamilton-Jacobi reachability analysis with deep learning, the framework identifies states that inevitably lead to failure, automatically generates labeled training data, and deploys a runtime monitor to trigger safety-preserving fallback mechanisms.

Introduction

Autonomous systems, such as self-driving cars and drones, increasingly rely on vision-based controllers. A significant challenge for these systems is handling OOD inputs. Traditional monitoring often focuses on individual components (e.g., image blurriness). However, the authors argue that system-level monitoring is more critical: an input is truly "anomalous" only if it leads to a violation of the robot's safety constraints within its specific closed-loop dynamics. This paper introduces a method to identify these system-level anomalies by analyzing the consequences of errors rather than just their magnitude.

System-Level Anomaly Detection Framework

The proposed framework operates in two distinct phases: offline training and online monitoring.

Overview of the offline training and online monitoring pipeline.
  1. Offline Failure Mining: Hamilton-Jacobi (HJ) Reachability analysis is used to compute a Backward Reachable Tube (BRT). The BRT defines the set of states from which a failure is guaranteed under the current vision-based controller.
  2. Automatic Data Labeling: A photorealistic simulator (X-Plane) renders images for states inside and outside the BRT, labeling them as "anomalous" or "safe," respectively. This eliminates the need for manual labeling of failure cases.
  3. Runtime Monitoring: A binary classifier (EfficientNet-B0) is trained on this dataset. During online operation, it flags anomalous inputs, prompting a switch to a fallback controller that prioritizes safety (e.g., by reducing aircraft velocity).

Experimental Evaluation

The framework was evaluated using the TaxiNet environment, which simulates an autonomous aircraft taxiing on a runway.

State space representation (a) and examples of varied visual conditions (b-g) used for training and testing.

The training set included 240,000 images across 12 conditions, including different times of day, weather, and runway markings. Testing involved unseen airports (KSFO, KEWR) and lighting conditions to evaluate the detector's generalization capabilities. The state space was defined by the aircraft's position relative to the centerline ($p_x, p_y$) and its heading angle ($\theta$).

Key Insights and Results

The study revealed that safety is highly context-dependent. For instance, certain positions were deemed anomalous in daylight but safe at night due to the high contrast provided by runway lights, which aided the controller's accuracy. This highlights that the BRT-based approach captures the actual consequences of visual errors.

Key results include:

  • High Reliability: The detector achieved accuracy between 89.7% and 98.1% and recall between 90.3% and 97.4% across various unseen environments.
  • Safety Improvement: The integrated pipeline reduced the volume of failure states from 25.75% down to 18.24%, significantly shrinking the system's "danger zone."
  • Real-time Performance: The monitor is lightweight enough to run at 25Hz, making it suitable for real-time deployment on robotic hardware.

Conclusion and Future Work

While effective, the current approach faces the "curse of dimensionality" in HJ Reachability, which typically struggles with high-dimensional state spaces. Future work aims to utilize sampling-based reachability methods (like DeepReach) and automate the semantic analysis of failure modes. Additionally, developing more sophisticated, "intelligent" fallback controllers remains a priority for enhancing autonomous resilience in complex environments.

Summary updated

BibTeX
@inproceedings{gupta2024detecting,
  author = {Aryaman Gupta and Kaustav Chakraborty and Somil Bansal},
  title = {{Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10611397},
}
Figure