A Framework for Robust Assimilation of Potentially Malign Third-Party Data, and Its Statistical Meaning

Matthew A. Wright, Roberto Horowitz

TL;DR

This paper introduces a novel framework for robustly estimating the state of dynamic systems, like traffic flow on freeways, by integrating statistical hypothesis testing into a particle filter. This allows the system to intelligently reject faulty or malicious sensor data, improving the reliability of state estimation, especially when dealing with uncertain third-party data sources like GNSS.

What Problem Does This Paper Address?

Intelligent Transportation Systems (ITS) heavily rely on real-time data from various sensors to operate effectively. However, this reliance makes them vulnerable to errors, faults, or even malicious manipulation of sensor data. Such "malign" data can lead to incorrect system states and, consequently, actively harmful control decisions. This is particularly true for third-party data sources like Global Navigation Satellite System (GNSS) data, which are increasingly used due to their low cost and widespread availability but are susceptible to various error sources (e.g., multipath propagation, signal blockage). Existing estimation methods, like standard particle filters, can diverge or produce numerical errors when faced with such unreliable data. The core problem this paper addresses is how to build estimation systems that can reliably distinguish between good and bad data, even when the nature of the faults is not fully known.

Methods

The authors extend the classic particle filter algorithm, a popular method for state estimation in nonlinear systems with noisy observations, by incorporating statistical hypothesis testing. The key innovation lies in using statistical tests to evaluate incoming sensor measurements against the system's observation model.

Two main statistical testing frameworks are adapted:

  • Neyman-Pearson Hypothesis Testing: This approach is used when a reliable model of potential data faults exists. The method constructs a likelihood ratio test to decide whether to accept or reject a measurement based on its probability under a "correct" model versus a "faulty" model.
  • Fisherian Significance Testing: This approach is employed when a specific fault model is unavailable. Instead of comparing against a fault model, it tests how well the observed data fits a null hypothesis (i.e., the data is correct). If the data significantly deviates from the null hypothesis, it's flagged as potentially faulty.

These statistical tests are integrated into the particle filter's update step. For each incoming measurement, a hypothesis test is performed. If a measurement is deemed faulty, it's rejected and not used in the particle filter's update. If it's accepted, it contributes to refining the state estimate. The authors also introduce Monte Carlo versions of these tests to make them computationally feasible within the particle filter framework.

Datasets

The paper uses a simulated freeway traffic scenario for evaluation. The simulation models a 19-mile stretch of I-210 West freeway using the Cell Transmission Model (CTM). The simulation incorporates:

  • Ground Truth: True density states for each link.
  • Simulated Measurements:
    • Noisy density measurements from 41 simulated loop detectors.
  • Simulated GNSS speed measurements from vehicles with a 2% penetration rate.
  • Fault Simulation: A 30% probability of faulty GNSS speed measurements was introduced, using two fault models:
  • A model for "stopped car" vehicles (near-zero speed).
  • A model for unrealistically fast movement or speeds matching the opposite freeway direction.
  • The "correct" model for velocity measurements was Gaussian with a mean of the true link velocity and a standard deviation of 20% of the mean.

No new benchmarks were introduced; the focus was on evaluating the proposed method on a realistic simulation of freeway traffic.

Key Insights and Ideas

  • Data vs. Model Trust: The framework allows a trusted system model to "outweigh" potentially erroneous data, a crucial capability for robust estimation, unlike standard particle filters that tend to accept all data.
  • Flexibility in Fault Modeling: The paper offers two distinct approaches (Neyman-Pearson and Fisherian) catering to situations where fault models are available versus when they are not, making the method broadly applicable.
  • Statistical Rigor for Data Rejection: Instead of ad-hoc outlier rejection, the paper grounds data rejection in established statistical theories, providing a principled way to handle unreliable sensor inputs.
  • The "Data" for Testing: An interesting nuance is that the "data" used for the hypothesis tests are the simulated particles themselves, which are derived from the system model. This means the test is essentially asking if the observations are consistent with the model's prediction of what good data should look like.

Key Results

The simulation results, presented in Tables I-III, demonstrate the effectiveness of the proposed methods:

  • Neyman-Pearson with Correct Fault Model (Table II): This approach achieved the best performance, significantly reducing labeling errors (false positives and false negatives) and achieving a Mean Absolute Percentage Error (MAPE) for density estimation close to the ideal scenario (3.51% vs. a baseline of 3.43% with no faulty data).
  • Neyman-Pearson with Incorrect Fault Model (Table I): This performed poorly, with high labeling errors, indicating the sensitivity to an inaccurate fault model.
  • Fisherian Significance Testing (Table III): This method showed performance that was generally between the two Neyman-Pearson scenarios. Crucially, for certain significance levels ($\alpha$), it outperformed the Neyman-Pearson method with an incorrect fault model. This highlights that a well-tuned Fisherian test can be effective even without a specific fault model.
  • Sensitivity to $\alpha$: The Fisherian approach's estimation accuracy was more sensitive to the chosen significance level ($\alpha$) compared to the Neyman-Pearson methods.

Limitations and Future Work

  • Computational Cost: While the Monte Carlo approach makes the tests feasible, integrating them into a particle filter still adds computational overhead.
  • Model Accuracy: The performance of the Neyman-Pearson test is highly dependent on the accuracy of the fault model.
  • Fisherian Complexity: The Fisherian approach, while model-free, can be more sensitive to the choice of $\alpha$ and might require careful tuning.
  • Future Work: The authors suggest exploring more "inductive" approaches for Fisherian testing, where the null hypothesis could be updated based on repeated measurements from the same sensor. They also mention that the formal extension to compound hypotheses (beyond simple-vs-simple) is a part of future work. The paper also hints at the potential for this framework to "overpower the data" with a highly trusted model, which could be further explored.

Summary updated

Abstract

This article presents a model-based method for fusing data from multiple sensors with a hypothesis-test-based component for rejecting potentially faulty or otherwise malign data. Our framework is based on an extension of the classic particle filter algorithm for real-time state estimation of uncertain systems with nonlinear dynamics with partial and noisy observations. This extension, based on classical statistical theories, utilizes statistical tests against the system's observation model. We discuss the application of the two major statistical testing frameworks, Fisher significance testing and Neyman?Pearson hypothesis testing, to the Monte Carlo and sensor fusion settings. The Monte Carlo Neyman?Pearson test we develop is useful when one has a reliable model of faulty data, while the Fisher method is applicable when one does not have a model of faults, which may occur when dealing with third-party data, such as the Global Navigation Satellite System (GNSS) data of transportation system users. These statistical tests can be combined with a particle filter to obtain a Monte Carlo state estimation scheme that is robust to faulty or outlier data. We present a synthetic freeway-traffic state estimation problem where the filters are able to reject simulated faulty GNSS measurements. The fault-model-free Fisher filter underperforms the Neyman?Pearson filter when the latter has an accurate fault model but outperforms it when the assumed fault model is incorrect.
BibTeX
@inproceedings{wright2020framework,
  author = {Matthew A. Wright and Roberto Horowitz},
  title = {{A Framework for Robust Assimilation of Potentially Malign Third-Party Data, and Its Statistical Meaning}},
  booktitle = {IEEE Intell. Transp. Syst. Mag},
  year = {2020},
  url = {https://openreview.net/forum?id=RE9VlkU-y6},
}
Figure