Efficient Out-of-Distribution Detection Using Latent Space of $β$-VAE for Cyber-Physical Systems

Shreyas Ramakrishna, Zahra Rahiminasab, Gabor Karsai, Arvind Easwaran, Abhishek Dubey

TL;DR

This paper introduces a highly efficient method for detecting when autonomous systems encounter "Out-of-Distribution" (OOD) scenarios—like extreme weather or unfamiliar roads—that they weren't trained to handle. By using a single $\beta$-Variational Autoencoder ($\beta$-VAE) with a partially disentangled latent space, the researchers can not only detect anomalies in real-time but also identify the specific environmental feature (e.g., heavy rain vs. high brightness) causing the issue, all while using significantly less memory and computation than traditional methods.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) are the "brains" of modern Cyber-Physical Systems (CPS), such as self-driving cars. However, these brains are only as good as their training data. If a car trained in sunny California suddenly encounters a torrential downpour in Southeast Asia, the DNN might make erroneous steering predictions, leading to catastrophic failures.

The authors identify two major challenges:

  1. The Safety Conundrum: DNNs are "black boxes." It is difficult to guarantee safety when the system operates in conditions (OOD) not covered during training.
  2. Efficiency vs. Reasoning: Existing OOD detectors often use "chains" of multiple classifiers—one for every possible feature like rain, fog, or lighting. This is computationally expensive and slow, making it impractical for resource-constrained hardware that needs to make split-second decisions.

Methods

The core innovation is the use of a $\beta$-Variational Autoencoder ($\beta$-VAE). Unlike a standard VAE, a $\beta$-VAE uses a hyperparameter ($\beta$) to force the model to learn a "disentangled" latent space. In a disentangled space, individual neurons (latent variables) correspond to specific, independent real-world features.

Fig. 2. Visualizing Latent Space Disentanglement: Scatter plots illustrating the latent distributions (𝜇, log(𝜎2)) generated using a 𝛽-VAE with different 𝛽values. Each latent distribution in the latent space is represented using distinct color shades. CARLA images generated in Section 5 was used to generate these latent distributions. For 𝛽=1, the generated latent distributions are entangled. For 𝛽> 1, the latent distributions are partially disentangled with a few latent distributions (inside the red circle) encoding independent features moving close to 𝜇=0 and log(𝜎2)=0, and the others moving away. Plot axis: x-axis represents the mean of the latent distributions in the range [-5,5], and the y-axis represents the log of variance in the range [-5,5].

The authors' pipeline includes:

  • Bayesian Optimization (BO): A heuristic to automatically find the optimal $\beta$ and number of latent variables ($n$) by maximizing the Mutual Information Gap (MIG), a metric for disentanglement.
  • Data Partitioning & Latent Mapping: The training data is grouped by feature variance. The researchers then use a KL-divergence-based heuristic to map specific latent variables to specific environmental features (e.g., identifying that Latent Variable #2 is highly sensitive to precipitation).
  • Runtime Detection: At runtime, the system monitors these specific latent variables. It uses Inductive Conformal Prediction (ICP) and Martingales to provide a robust statistical measure of how much the current scene deviates from the training set.

Datasets

The approach was validated on two distinct datasets:

  1. CARLA Simulator: A high-fidelity autonomous driving simulator used to generate synthetic scenes with controlled variations in precipitation, brightness, and road segments.
  2. nuImages: A large-scale, real-world automotive dataset. The authors used a fragment of this to test their detector on complex, real-world features like traffic density and pedestrian presence.
Fig. 1. Scenes and feature representations: (a) scenes with feature labels from CARLA simulation, (b) hierarchical representation of an image using its features. The features can take discrete or continuous values.

Key Insights and Ideas

  • Partial Disentanglement is Enough: While full disentanglement is notoriously difficult for complex real-world data, the authors show that partial disentanglement—where only the most informative features are separated—is sufficient for robust OOD detection and reasoning.
  • Reasoning via Latent Sensitivity: By knowing which latent variable "belongs" to which feature, the system can explain why it is failing. If the "precipitation neuron" spikes, the system knows the OOD condition is due to rain, allowing it to switch to a safer controller (like one based on Radar instead of Cameras).
  • Efficiency through Sparsity: Instead of monitoring thousands of activation functions or running dozens of separate classifiers, this method only needs to monitor a handful of informative latent variables (e.g., 5 variables instead of 1024).

Key Results

  • High Accuracy: The $\beta$-VAE detector achieved a 96.98% F1-score (robustness) and 96% minimum sensitivity in CARLA simulations.
  • Massive Memory Savings: For a real-world dataset like nuScenes (with 38 labels), a traditional chain of classifiers would require ~136.8 GB of memory. The proposed single-network approach requires only 10.96 GB—a 92% reduction.
  • Real-Time Performance: The detector's inference time was approximately 74ms, well within the typical 100ms sampling period required for autonomous vehicle safety.
  • Effective Reasoning: In "New Road" scenarios, the system correctly identified that the scene was OOD but not due to rain or brightness, accurately pointing to an "out-of-set" feature (the new road geometry).

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Manual Thresholding: Currently, some data partitioning steps require human-selected thresholds, which they aim to automate.
  • Feature Correlation: In the real world, features are often correlated (e.g., rain often comes with lower brightness). This can make mapping a single latent variable to a single feature more difficult.
  • Complex Reasoning: While the system can identify if a known feature is the cause, reasoning about completely novel, "out-of-set" features remains a challenge for future research.

Summary updated

Abstract

Deep Neural Networks are actively being used in the design of autonomous Cyber-Physical Systems (CPSs). The advantage of these models is their ability to handle high-dimensional state-space and learn compact surrogate representations of the operational state spaces. However, the problem is that the sampled observations used for training the model may never cover the entire state space of the physical environment, and as a result, the system will likely operate in conditions that do not belong to the training distribution. These conditions that do not belong to training distribution are referred to as Out-of-Distribution (OOD). Detecting OOD conditions at runtime is critical for the safety of CPS. In addition, it is also desirable to identify the context or the feature(s) that are the source of OOD to select an appropriate control action to mitigate the consequences that may arise because of the OOD condition. In this paper, we study this problem as a multi-labeled time series OOD detection problem over images, where the OOD is defined both sequentially across short time windows (change points) as well as across the training data distribution. A common approach to solving this problem is the use of multi-chained one-class classifiers. However, this approach is expensive for CPSs that have limited computational resources and require short inference times. Our contribution is an approach to design and train a single $β$-Variational Autoencoder detector with a partially disentangled latent space sensitive to variations in image features. We use the feature sensitive latent variables in the latent space to detect OOD images and identify the most likely feature(s) responsible for the OOD. We demonstrate our approach using an Autonomous Vehicle in the CARLA simulator and a real-world automotive dataset called nuImages.
BibTeX
@article{ramakrishna2022efficient,
  author = {Shreyas Ramakrishna and Zahra Rahiminasab and Gabor Karsai and Arvind Easwaran and Abhishek Dubey},
  title = {{Efficient Out-of-Distribution Detection Using Latent Space of $β$-VAE for Cyber-Physical Systems}},
  journal = {ACM Trans. Cyber Phys. Syst.},
  year = {2022},
  doi = {10.1145/3491243},
}
Figure