Missing Traffic Data Imputation with a Linear Generative Model Based on Probabilistic Principal Component Analysis

Liping Huang, Zhenghuan Li, Ruikang Luo, Rong Su

TL;DR

This paper introduces a novel linear generative model based on Probabilistic Principal Component Analysis (PPCA) to address the challenging problem of "missing not at random" (MNAR) traffic data. They also propose a "p-score" metric to identify which road links are more critical for imputation, finding that links with higher p-scores are better recovered.

What Problem Does This Paper Address?

Intelligent transportation systems rely heavily on traffic data for various applications like forecasting and control. However, this data is often incomplete due to sensor malfunctions or limited coverage by mobile sensors. While many methods exist for imputing missing data, they typically assume data is "missing at random" (MAR). In reality, traffic data is often "missing not at random" (MNAR), meaning the missingness is related to the unobserved values themselves, making imputation much harder. Furthermore, existing methods don't account for the fact that different road links might have varying importance or connectivity, which could affect imputation accuracy. This paper tackles both the MNAR challenge and the differential importance of road links for imputation.

Methods

The core of this work is a linear generative model built upon Probabilistic Principal Component Analysis (PPCA). PPCA is a statistical method that models data as being generated from a lower-dimensional latent space. The authors adapt this to traffic data, assuming traffic speed observations are a linear combination of latent variables with added noise.

A key innovation is the introduction of a "p-score" metric. This metric quantifies the importance of a road link by measuring the average Pearson correlation of its time series data with all other links in the network. The intuition is that links with higher p-scores are more interconnected and thus potentially more critical for understanding overall traffic dynamics. This p-score is then used to analyze how well the imputation method performs on links with different levels of connectivity.

The paper also details the mathematical derivations for estimating model parameters, including mean and covariance, under the PPCA framework, specifically addressing the MNAR scenario.

Datasets

The authors evaluated their proposed method using a real-world traffic speed dataset collected from urban expressways in China. This dataset covers two months (August to September 2016) and includes speed data for twenty selected road links during morning rush hours (7:00 AM to 9:00 AM). The data was preprocessed into a "congestion index" and normalized to follow a Gaussian distribution, a key assumption of the PPCA model.

Key Insights and Ideas

  • MNAR is a critical challenge: The paper highlights that real-world traffic data is often MNAR, and existing methods struggle with this. Their model specifically targets this.
  • Link importance matters: Not all road links are created equal when it comes to imputation. The p-score metric provides a data-driven way to identify which links are more central to the network's traffic patterns.
  • Higher connectivity, better imputation: A significant finding is that road links with higher p-scores (indicating stronger correlations with other links) tend to have their missing data imputed more accurately. This suggests that understanding network topology is crucial for effective imputation.
  • PPCA as a strong baseline: The paper leverages PPCA, a well-established technique for handling missing data, and builds upon it to address the specific challenges of MNAR traffic data.

Key Results

The proposed PPCA-based model consistently outperformed the standard "ppca-em" (Expectation-Maximization PPCA) model across various metrics, including Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Symmetric Mean Absolute Percentage Error (SMAPE), and R². For instance, in one experiment setting, the new model achieved an accuracy of 88.9% compared to 83.0% for ppca-em.

Crucially, the paper demonstrated that missing data on links with higher p-scores was recovered with better accuracy. For example, link 18 (with a p-score of 14.00) showed significantly better imputation results than link 6 (with a p-score of 4.85) across different missing data percentages. The model also showed improved computational efficiency, with significantly lower computing times compared to the baseline.

Limitations and Future Work

The authors acknowledge that their model is a linear generative model. While this offers interpretability and efficiency, it might not capture highly complex, non-linear traffic dynamics. They suggest future work could involve comparing their model with other methods on more diverse traffic datasets and exploring more advanced imputation techniques. The study also focuses on specific road links during rush hour, and its generalizability to other times or network structures could be further investigated.

Summary updated

Abstract

Even with the ubiquitous sensing data in intelligent transportation systems, such as the mobile sensing of vehicle trajectories, traffic estimation is still faced with the data missing problem due to the detector faults or limited number of probe vehicles as mobile sensors. Such data missing issue poses an obstacle for many further explorations, e.g., the link-based traffic status modeling. Although many studies have focused on tackling this kind of problem, existing studies mainly focus on the situation in which data are missing at random and ignore the distinction between links of missing data. In the practical scenario, traffic speed data are always missing not at random (MNAR). The distinction for recovering missing data on different links has not been studied yet. In this paper, we propose a general linear model based on probabilistic principal component analysis (PPCA) for solving MNAR traffic speed data imputation. Furthermore, we propose a metric, i.e., Pearson score (p-score), for distinguishing links and investigate how the model performs on links with different p-score values. Experimental results show that the new model outperforms the typically used PPCA model, and missing data on links with higher p-score values can be better recovered.
BibTeX
@inproceedings{huang2022missing,
  author = {Liping Huang and Zhenghuan Li and Ruikang Luo and Rong Su},
  title = {{Missing Traffic Data Imputation with a Linear Generative Model Based on Probabilistic Principal Component Analysis}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s23010204},
}
Figure