A Framework for Robust Assimilation of Potentially Malign Third-Party Data, and its Statistical Meaning

Matthew A. Wright, Roberto Horowitz

Abstract

This paper presents a model-based method for fusing data from multiple sensors with a hypothesis-test-based component for rejecting potentially faulty or otherwise malign data. Our framework is based on an extension of the classic particle filter algorithm for real-time state estimation of uncertain systems with nonlinear dynamics with partial and noisy observations. This extension, based on classical statistical theories, utilizes statistical tests against the system's observation model. We discuss the application of the two major statistical testing frameworks, Fisherian significance testing and Neyman-Pearsonian hypothesis testing, to the Monte Carlo and sensor fusion settings. The Monte Carlo Neyman-Pearson test we develop is useful when one has a reliable model of faulty data, while the Fisher one is applicable when one may not have a model of faults, which may occur when dealing with third-party data, like GNSS data of transportation system users. These statistical tests can be combined with a particle filter to obtain a Monte Carlo state estimation scheme that is robust to faulty or outlier data. We present a synthetic freeway traffic state estimation problem where the filters are able to reject simulated faulty GNSS measurements. The fault-model-free Fisher filter, while underperforming the Neyman-Pearson one when the latter has an accurate fault model, outperforms it when the assumed fault model is incorrect.

Introduction

Modern Intelligent Transportation Systems (ITS) increasingly rely on third-party data, such as GPS (GNSS) speeds from connected vehicles, to estimate traffic conditions. However, this data can be highly unreliable due to sensor noise, multipath errors, or even malicious spoofing attacks. Standard state estimation algorithms, like particle filters, are fragile when fed highly improbable data. A few bad data points—for example, a stopped car incorrectly reporting its location on a free-flowing highway—can cause the state estimate to diverge entirely or crash the software due to numerical underflow. This research addresses the challenge of identifying and rejecting this malign data in real-time, particularly in scenarios where engineers lack a reliable mathematical model of the faulty data.

Methodology

The proposed framework extends the classic particle filter—a Monte Carlo-based algorithm for real-time state estimation in complex, nonlinear systems—by introducing a hypothesis-testing step before the filter's update phase. Two distinct approaches are developed based on classical statistics:

  • Monte Carlo Neyman-Pearson Test: Applied when a reliable model for faulty data is available. It performs a likelihood ratio test between a null hypothesis (normal sensor behavior) and an alternative hypothesis (faulty behavior).
  • Monte Carlo Fisherian Test: Applied when no model for faults exists. It calculates an empirical p-value based solely on the null hypothesis (expected normal behavior). If the incoming measurement is statistically improbable given the system's current state estimate, it is rejected.

A novel aspect of this approach is the inversion of standard statistical testing: simulated particles (representing the trusted physical model) act as the "data" distribution, and physical sensor measurements are evaluated against this distribution for acceptance or rejection.

Experimental Setup

The framework was evaluated using a simulated traffic environment on a 19-mile stretch of I-210 West in Southern California, modeled using the macroscopic Cell Transmission Model (CTM). The sensor suite included simulated noisy density measurements from 41 inductive loop detectors and vehicle-carried GNSS transponders measuring speed at a 2% penetration rate. To test robustness, 30% of the GNSS speed measurements were intentionally corrupted, for instance, by reporting zero speed to simulate a stopped car or drawing from a random high-speed Gaussian distribution.

True density state and speed measurements for the I-210 West simulation.

Results and Discussion

The experiments demonstrate that the Fisherian approach can effectively reject anomalies without a prior fault model by rigorously defining "normal" behavior using the system's dynamic model. While a Neyman-Pearson filter with a correct fault model achieved the lowest Mean Absolute Percentage Error (MAPE) of 3.53%, its performance degraded to 3.86% when the fault model was incorrect. In contrast, the Fisherian filter achieved a robust 3.66% MAPE (at $\alpha = 0.001$), outperforming the Neyman-Pearson filter that relied on incorrect assumptions.

Performance metrics for the Fisherian filter across different significance levels.

Key insights include:

  • Model Robustness: A bad fault model is worse than no fault model. The Fisherian filter provides a reliable fallback when fault characteristics are unknown.
  • Model-Data Balance: The framework allows a highly-trusted physical model to overrule and reject clear outliers in observational data, bridging the gap between model-based engineering and data-driven estimation.

Conclusion

This robust particle filtering framework provides a statistically grounded method for assimilating third-party data while maintaining system stability against malign inputs. While the current Fisherian implementation uses a hard significance threshold ($\alpha$), future work will explore more inductive approaches that update the null hypothesis over time and handle compound hypotheses to further improve estimation accuracy in dynamic environments.

Summary updated

Abstract

This paper presents a model-based method for fusing data from multiple sensors with a hypothesis-test-based component for rejecting potentially faulty or otherwise malign data. Our framework is based on an extension of the classic particle filter algorithm for real-time state estimation of uncertain systems with nonlinear dynamics with partial and noisy observations. This extension, based on classical statistical theories, utilizes statistical tests against the system's observation model. We discuss the application of the two major statistical testing frameworks, Fisherian significance testing and Neyman-Pearsonian hypothesis testing, to the Monte Carlo and sensor fusion settings. The Monte Carlo Neyman-Pearson test we develop is useful when one has a reliable model of faulty data, while the Fisher one is applicable when one may not have a model of faults, which may occur when dealing with third-party data, like GNSS data of transportation system users. These statistical tests can be combined with a particle filter to obtain a Monte Carlo state estimation scheme that is robust to faulty or outlier data. We present a synthetic freeway traffic state estimation problem where the filters are able to reject simulated faulty GNSS measurements. The fault-model-free Fisher filter, while underperforming the Neyman-Pearson one when the latter has an accurate fault model, outperforms it when the assumed fault model is incorrect.
BibTeX
@inproceedings{wright2020framework,
  author = {Matthew A. Wright and Roberto Horowitz},
  title = {{A Framework for Robust Assimilation of Potentially Malign Third-Party Data, and its Statistical Meaning}},
  booktitle = {IEEE Intell. Transp. Syst. Mag},
  year = {2020},
  url = {https://openreview.net/forum?id=RE9VlkU-y6},
}

Loading related content...

Figure