A Spatial Missing Value Imputation Method for Multi-view Urban Statistical Data

Yongshun Gong, Zhibin Li, Jian Zhang, Wei Liu, Bei Chen, Xiangjun DongIJCAI 2020 · DOI

TL;DR

This paper introduces SMV-NMF, a novel method for imputing missing values in multi-view urban statistical data by integrating spatial relationships and multi-view learning. It addresses the challenge of incomplete urban data, which hinders accurate analysis, by leveraging both regional similarities and attribute correlations to achieve superior imputation performance.

What Problem Does This Paper Address?

Urban statistical data, encompassing views like population, economy, and family, are crucial for understanding city development and informing policy. However, this data often suffers from missing values due to various issues like document defacement or recording errors. These missing values can distort analyses and lead to flawed conclusions. Existing imputation methods often struggle with the unique challenges of urban data: missing values are frequently not random (e.g., missing in the past and present), data has multiple interconnected "views" (attributes), and spatial proximity significantly influences data values. This paper aims to fill this gap by proposing a method that effectively handles these multi-view, spatially correlated, and non-random missing data problems.

Methods

The core of the proposed method, SMV-NMF, is built upon Non-negative Matrix Factorization (NMF) but enhanced with several key innovations:

  • Spatially related Multi-Kernel K-Means (S-MKKM): This method is used to identify underlying relationships among multiple views and capture regional similarities. It leverages a graph Laplacian dynamics strategy to cluster regions into functional groups, reflecting how similar functional properties (e.g., business districts) can exist in geographically close or functionally related areas.
  • Adaptive-Weight NMF: To improve the reliability of imputation, an adaptive weighting strategy is introduced. This strategy uses an exponential function based on the distance between a region's latent representation and its cluster centroid to assign weights, effectively controlling how much information from the guidance matrices is used.
  • Single-view and KNN Guidances: The method also incorporates guidance from individual views using a spatially related K-Means (S-KKM) approach. Additionally, it leverages K-Nearest Neighbors (KNN) to capture local spatial information, creating an imputed matrix (Xknn) that reflects the average values of neighboring regions.
  • Combined Optimization: The final objective function combines the original NMF objective with terms for the spatial guidance (Xmv), single-view guidance (Xs), and KNN guidance (Xknn), all weighted by regularization parameters. Multiplicative update rules are used to ensure convergence.

Datasets

The authors evaluated their method on six real-world urban statistical datasets from Australian cities:

  • Sydney
  • Melbourne
  • Brisbane
  • Perth
  • SYD-large (larger scale)
  • MEL-large (larger scale)

Each dataset contains four views: Economy, Family, Income, and Population. The datasets vary in size, with the "-large" versions containing significantly more fine-grained regions. The authors note that while these datasets are from Australian cities, their method is generalizable to other urban areas.

Key Insights and Ideas

  • Spatial relationships are paramount: The paper strongly emphasizes that simply treating missing data as random or ignoring spatial proximity leads to suboptimal imputation. The "First Law of Geography" (near things have more correlations than distant things) is a guiding principle.
  • Multi-view data is interconnected: Ignoring correlations between different data views (e.g., economy and population) can lead to poor imputation. The method explicitly models these interdependencies.
  • Non-random missingness is a challenge: The paper highlights that missing data in urban statistics often follows patterns (e.g., missing in past and present), which violates assumptions of standard matrix completion methods.
  • Functional similarity over pure distance: Figure 1 illustrates that regions can be functionally similar (e.g., business centers) even if not physically closest, and this functional similarity is crucial for imputation.
  • Generalizability is achievable: The proposed SMV-NMF demonstrates strong generalizability, meaning a model trained on one city's data can perform well on another, suggesting underlying commonalities in urban data structures.

Key Results

  • SMV-NMF consistently outperforms 12 baseline methods across all six datasets and various missing ratios.
  • The paper reports significant improvements in Mean Relative Error (MRE) and Root Mean Square Error (RMSE) compared to existing methods. For instance, in Table 1, SMV-NMF achieves the lowest RMSE and MRE across multiple datasets.
  • The generalizability tests (Figure 5 and Table 2) show that SMV-NMF can effectively transfer models between different urban datasets, outperforming other methods like MVL-IV in this regard.
  • Ablation studies (e.g., removing the non-negativity constraint or spatial guidance) demonstrate the effectiveness of each component of SMV-NMF.

Limitations and Future Work

  • Computational Complexity: The paper acknowledges that the pursuit of pinpoint accuracy leads to a higher computational complexity per iteration (O(nk²)), sacrificing some efficiency.
  • Fixed Parameters: For practical reasons and due to the intrinsic properties of urban statistical data, certain parameters (like the number of clusters 'l' and regularization parameter 'a') were fixed in the experiments. Future work could explore adaptive parameter tuning.
  • Temporal Data: While the paper focuses on static spatial data, it mentions that spatio-temporal methods exist but don't perform well with missing temporal information. Extending SMV-NMF to handle spatio-temporal data could be a future direction.
  • Further Exploration of Guidance: The paper suggests that further exploration of guidance matrices and their integration could lead to even better imputation performance.

Summary updated

BibTeX
@inproceedings{gong2020spatial,
  author = {Yongshun Gong and Zhibin Li and Jian Zhang and Wei Liu and Bei Chen and Xiangjun Dong},
  title = {{A Spatial Missing Value Imputation Method for Multi-view Urban Statistical Data}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/182},
}
Figure