TL;DR
This paper benchmarks various regression models for spatial data, particularly focusing on how well they handle spatial heterogeneity (where relationships vary across space). They introduce a synthetic data framework to systematically test models and propose a decision tree to guide model selection based on data characteristics like sample density and expected linearity.
What Problem Does This Paper Address?
The increasing use of machine learning (ML) in GIScience has led to the application of global ML models to spatial data without explicitly accounting for spatial structure. While spatial statistics methods have long addressed spatial autocorrelation and heterogeneity, there's a gap in understanding when these specialized methods are truly necessary compared to standard ML approaches. This paper aims to fill this gap by systematically benchmarking different regression models (both global ML and local spatial models) under varying degrees of spatial heterogeneity, helping researchers choose the most appropriate model for their specific spatial data problem.
Methods
The authors developed a synthetic data-generating process (DGP) that allows for systematic variation of:
- Spatial Non-stationarity: The dependence of the target variable on input features varies across space. This was modeled by adding a spatially varying component to regression coefficients.
- Error Heterogeneity: The variance of the error term also varies across space, either following a trigonometric pattern or a linear trend.
- Signal-to-Noise Ratio: Controlled by the overall noise level.
- Data Generating Process: Both linear and non-linear relationships were simulated.
They benchmarked seven regression models:
- Global Linear Models: Ordinary Least Squares (OLS) and Spatial Lag in X (SLX).
- Local Linear Model: Geographically Weighted Regression (GWR).
- Global Non-linear Models: Random Forest (RF) and RF with coordinates as input.
- Local Non-linear Models: Spatial Random Forest (spatial RF).
- Hybrid Model: Regression Kriging (using RF as the base regressor).
The models were evaluated based on their Root Mean Squared Error (RMSE) on unseen test data.
Datasets
The paper uses two types of datasets:
- Synthetic Datasets: Generated using the novel DGP described above, allowing for controlled experiments with varying degrees of spatial non-stationarity, error heterogeneity, signal-to-noise ratio, and sample size (n=100, 500, 1000, 5000).
- Real-World Datasets: Five publicly available datasets were used for validation:
- California housing dataset
- Atlantic mortality dataset
- Deforestation rates dataset
- Meuse river dataset
- Plant richness dataset
Key Insights and Ideas
- Overfitting is a Real Risk, Even for Linear Local Models: The study highlights that even linear local models like GWR can overfit, especially with limited sample sizes or when dealing with noise and non-stationarity. This underscores the importance of evaluating models on independent test data, not just training data fit.
- Local Models Shine with Small Samples and High Heterogeneity: GWR and other local models demonstrate advantages when sample sizes are small and spatial heterogeneity is high, particularly for linear relationships.
- Non-linear Models for Non-linear Phenomena: Random Forests generally outperform linear models for non-linear data generating processes, especially with higher sample densities.
- Spatial RFs Don't Always Add Value: The spatial versions of Random Forests (spatial RF and RF with coordinates) did not consistently outperform global RFs in the tested scenarios, sometimes even showing worse performance, possibly due to overfitting local models on limited data or issues with spatial generalization.
- Noise Pattern Matters for Linear DGPs: The distribution of error variance can significantly impact model performance on linear data, especially when it mirrors the spatial non-stationarity pattern. For non-linear DGPs, this effect is less pronounced.
Key Results
- Linear DGP: GWR showed superior performance in handling spatial heterogeneity compared to OLS and SLX.
- Non-linear DGP: Random Forest-based models generally performed better than linear models.
- Sample Size: With high sample density, RF models were competitive even on linear data. For very low sample densities, linear models (like GWR) were more advisable.
- Real-World Data: The proposed model selection criteria were validated. GWR performed well on datasets with lower heterogeneity and smaller sample sizes (Atlantic, Meuse), while RF-based methods were better for datasets with higher sample density and complexity (California housing, deforestation, plants).
Limitations and Future Work
- Limited Scope: The analysis focused on specific types of heterogeneity and models. Future work could explore spatial autocorrelation's interplay with non-stationarity, and other non-linear relations.
- Integration of Advanced ML: The authors suggest integrating modern ML models like spatial neural networks with geospatial principles.
- Big Data: The current analysis does not cover "big data" scenarios, where different memory-efficient methods might be needed.
- A Priori Stationarity Assessment: There's a need for better methods to assess spatial stationarity before model selection.
- Model Interpretability: While prediction is key for validation, interpretability is also crucial for model selection and might favor linear models in some cases. Future work could aim to improve the spatial interpretability of global models like RFs.
Summary updated