Vector Autoregressive Model-Based Anomaly Detection in Aviation Systems

Igor Melnyk, Bryan L. Matthews, Hamed Valizadegan, Arindam Banerjee, Nikunj C. OzaJ. Aerosp. Inf. Syst 2016 · OpenReview

TL;DR

This paper introduces a novel framework for detecting anomalies in aviation systems by modeling flight data as a Semi-Markov Switching Vector Autoregressive (SMS-VAR) process. By analyzing the discrepancies between predicted and observed flight phases, the method effectively identifies operationally significant events that could indicate safety risks, outperforming existing approaches on various anomaly types.

What Problem Does This Paper Address?

The increasing volume of air traffic necessitates robust systems for detecting anomalies in flight data to ensure aviation safety. Existing methods often struggle with the heterogeneous, multivariate, and variable-length nature of flight data, which includes sensor readings and pilot switch inputs. These methods can be limited by their inability to capture complex interdependencies between features or their reliance on predefined thresholds, making them prone to missing novel or subtle anomalies. The paper aims to address this by developing a framework that can effectively model and detect anomalies in such complex datasets, identifying potential mechanical, environmental, or human factors contributing to safety risks.

Methods

The core of the proposed framework is the Semi-Markov Switching Vector Autoregressive (SMS-VAR) model. This model represents each flight as a sequence of hidden "phases" (unobserved states) and observed "modes" (pilot switch configurations).

  • SMS-VAR Model: This model combines:
  • Vector Autoregressive (VAR) processes: To model the continuous sensor measurements within each phase. Each phase has its own distinct VAR model.
  • Semi-Markov Models (SMM): To capture the switching behavior between different flight modes (pilot switch combinations) and their durations. This is an extension of standard Markov models, allowing for non-geometric state durations.
  • Hidden Phase Variable (x): An unobserved variable that groups consecutive mode states into distinct flight phases, allowing for longer-term consistency within a phase despite mode changes.
  • Anomaly Detection: The anomaly detection is based on comparing the predicted phase distribution (based on the SMS-VAR model) with the filtered phase distribution (after observing the actual flight data).
  • A Kullback-Leibler (KL) divergence is used to measure the difference between these distributions at each time step.
  • The standard deviation of these KL divergence values over the entire flight is used as the anomaly score. Higher standard deviations indicate more anomalous flights.

Datasets

  • Simulated Datasets: Three synthetic datasets were generated to test the detection of different anomaly types:
  • Mode anomalies (unusual switch flips).
  • Phase anomalies (unusual changes in underlying dynamics).
  • Sensor anomalies (errors in continuous measurements). These datasets consisted of 100 normal and 10 anomalous flights, each 200 time stamps long.

  • Real Flight Data:

    • A dataset of 20,000 unlabeled flights from a partner airline company was used. This data contained approximately 300 parameters sampled at 1 Hz, from which 31 continuous and 18 discrete binary parameters were selected.
  • A smaller, manually labeled dataset of 10 go-around flights and 100 normal flights was also used for evaluation.

Key Insights and Ideas

  • Hierarchical Modeling for Flight Data: The SMS-VAR model's ability to capture both discrete (pilot switches/modes) and continuous (sensor data) aspects of flight, organized into phases, is crucial for understanding complex aviation dynamics.
  • Phase-Based Anomaly Detection: Focusing on the discrepancy between predicted and observed phase distributions offers a more sensitive measure of anomaly than simple likelihood or prediction error. This allows for the detection of subtle deviations in flight behavior.
  • Semi-Markov for Realistic Durations: Using semi-Markov models for mode durations is more appropriate for aviation than simple Markov models, as flight phases and modes can persist for varying, non-geometric lengths of time.
  • Complementarity of Approaches: The paper suggests that SMS-VAR can complement existing methods by detecting a broader spectrum of anomalies, particularly those involving short-duration events or complex interactions between discrete and continuous data.

Key Results

  • High Detection Accuracy: The SMS-VAR KL divergence-based approach demonstrated high accuracy in detecting various simulated anomaly types (mode, phase, and sensor anomalies), often outperforming baseline methods like VAR and SMM.
  • Superiority on Real Data: On both labeled and unlabeled real flight data, SMS-VAR KL consistently achieved superior or comparable performance to other methods, particularly in identifying go-around events and unusual pilot switch changes.
  • Robustness to Anomaly Proportion: The method showed good detection accuracy even with a moderate proportion of anomalous flights (up to 15%), though accuracy decreased significantly when anomalies constituted 50% or more of the dataset, as the model then started representing anomalous behavior.
  • Effective for Short-Term Anomalies: The standard deviation of KL divergence proved effective in identifying short-duration anomalies, a common characteristic of many aviation safety events.

Limitations and Future Work

  • Model Complexity: While the framework is designed to be scalable, the parameter learning (EM algorithm) can be computationally challenging, especially with very long time series.
  • Assumption on Anomaly Proportion: The model's performance degrades when the proportion of anomalous flights is high, as the learned model may then represent anomalous behavior.
  • Future Work: The authors plan to extend the anomaly detection approach to include the ability to model long-duration anomaly events, which are not as effectively captured by the current short-memory SMS-VAR model.

Summary updated

Abstract

Detecting anomalies in datasets, where each data object is a multivariate time series, possibly of different length for each data object, is emerging as a key problem in certain domains. The problem is considered in the context of aviation safety, where data objects are flights of various durations and the multivariate time series corresponds to sensor readings. The goal is then to detect anomalous flight segments, due to mechanical, environmental, or human factors. In this paper, a general framework is presented for anomaly detection in such settings by representing each multivariate time series using a vector autoregressive exogenous model, constructing a distance matrix among the objects based on their respective vector autoregressive exogenous models, and finally detecting anomalies based on the object dissimilarities. The framework is scalable, due to the inherent parallel nature of most computations, and can be used to perform online anomaly detection. Experimental results on a real flight dataset illustrate that the framework can detect different types of multivariate anomalies along with the key parameters involved.
BibTeX
@article{melnyk2016vector,
  author = {Igor Melnyk and Bryan L. Matthews and Hamed Valizadegan and Arindam Banerjee and Nikunj C. Oza},
  title = {{Vector Autoregressive Model-Based Anomaly Detection in Aviation Systems}},
  journal = {J. Aerosp. Inf. Syst.},
  year = {2016},
  url = {https://openreview.net/forum?id=5qbre3wiTE0},
}
Figure