Validation approaches for satellite-based PM2.5 estimation: Assessment and a new approach

Tongwen Li, Huanfeng Shen, Qiangqiang Yuan, Liangpei ZhangarXiv 2018 · DOI · arXiv

Abstract

Satellite-derived aerosol optical depth (AOD) has been increasingly employed for the estimation of ground-level PM2.5, which is often achieved by modeling the relationship between AOD and PM2.5. To evaluate the accuracy of PM2.5 estimation, the cross-validation (CV) technique has been widely used. There have been several CV-based validation approaches applied for the AOD-PM2.5 models. However, the applicable conditions of these validation approaches still remain unclear. Additionally, is there space to develop better validation approaches for the AOD-PM2.5 models? The contributions of this study can be summarized as two aspects. Firstly, we comprehensively analyze and assess the existing validation approaches, and give the suggestions for applicable conditions of them. Then, the existing validation approaches do not take the distance to monitoring station into consideration. A new validation approach considering the distance to monitoring station is proposed in this study. Among the existing validation approaches, the sample-based CV is used to reflect the overall prediction ability; the site-based CV and the region-based CV have the potentials to evaluate spatial prediction performance; the time-based CV and the historical validation are capable of evaluating temporal prediction accuracy. In addition, the validation results indicate that the proposed validation approach has shown great potentials to better evaluate the accuracy of PM2.5 estimation. This study provides application implications and new perspectives for the validation of AOD-PM2.5 models.

Introduction

Ground-level fine particulate matter ($PM_{2.5}$) is a critical environmental health indicator. Due to the sparse distribution of ground monitoring stations, researchers frequently use satellite-derived Aerosol Optical Depth (AOD) to estimate $PM_{2.5}$ concentrations across large areas. However, the validation of these models often lacks consistency, leading to potentially misleading accuracy reports. This study assesses existing validation methods and proposes a new distance-aware approach to better evaluate model performance in "data deserts."

Categorizing Existing Validation Strategies

Cross-validation (CV) is the standard for assessing AOD-$PM_{2.5}$ models, yet its application varies significantly across the literature. The researchers categorized five primary strategies used to partition data for testing:

  1. Sample-based CV: Randomly hides individual data points to reflect overall prediction ability.
  2. Site-based CV: Hides all data from specific stations to evaluate spatial prediction performance.
  3. Region-based CV: Hides data from entire geographic areas.
  4. Time-based CV: Hides specific time periods to test temporal prediction.
  5. Historical Validation: Uses past data to predict future concentrations.
Schematic of common cross-validation (CV) and validation approaches for AOD-PM2.5 models.

Each approach serves a different purpose but also carries specific limitations. For instance, site-based CV often only reflects accuracy near existing stations, while sample-based CV may suffer from data leakage where training and validation sets contain the same stations, artificially inflating performance metrics.

Comparison of existing validation approaches, their applicable conditions, and limitations.

The Distance-Aware Validation Approach

A significant gap in current methods is the failure to account for spatial proximity. Monitoring stations are often clustered in urban areas. In a standard site-based CV, a "hidden" validation station might be very close to a "visible" training station, creating a proximity bias that makes models look more accurate than they actually are when predicting for remote areas.

To address this, the authors proposed a distance-aware validation method. In this approach, for every validation station, any training stations within a specified distance ($d$) are excluded. By varying $d$ from 0 to 110 km, the study quantifies how model accuracy decays as the distance from known data points increases.

Key Insights and Results

The researchers tested this approach using a Geo-Intelligent Deep Belief Network (Geoi-DBN) on 2015 data from China. The results revealed a "two-stage" decay in model performance:

  • Stage 1 (0–30 km): Accuracy drops sharply ($R^2$ decline slope of -0.0062), indicating that local spatial correlations are strong but lose their predictive power quickly.
  • Stage 2 (30–110 km): Accuracy continues to decline but at a much slower, steadier rate (-0.0015).

Quantitatively, the model's $R^2$ dropped from 0.84 (at 0 km, standard site-based CV) to 0.58 at 110 km. Correspondingly, the Root Mean Square Error (RMSE) rose from 15.39 $\mu g/m^3$ to 24.90 $\mu g/m^3$. This demonstrates that standard validation methods significantly overestimate model performance in remote, unmonitored regions.

Conclusion

This study provides a roadmap for more rigorous model validation in air quality research. While sample-based CV is useful for general performance, site-based or the new distance-aware approach should be used to assess spatial reliability. The proposed distance-aware method offers a more honest assessment of air quality mapping in areas far from monitoring networks, providing critical insights for environmental policy and public health monitoring in "data deserts."

Summary updated

Abstract

Satellite-derived aerosol optical depth (AOD) has been increasingly employed for the estimation of ground-level PM2.5, which is often achieved by modeling the relationship between AOD and PM2.5. To evaluate the accuracy of PM2.5 estimation, the cross-validation (CV) technique has been widely used. There have been several CV-based validation approaches applied for the AOD-PM2.5 models. However, the applicable conditions of these validation approaches still remain unclear. Additionally, is there space to develop better validation approaches for the AOD-PM2.5 models? The contributions of this study can be summarized as two aspects. Firstly, we comprehensively analyze and assess the existing validation approaches, and give the suggestions for applicable conditions of them. Then, the existing validation approaches do not take the distance to monitoring station into consideration. A new validation approach considering the distance to monitoring station is proposed in this study. Among the existing validation approaches, the sample-based CV is used to reflect the overall prediction ability; the site-based CV and the region-based CV have the potentials to evaluate spatial prediction performance; the time-based CV and the historical validation are capable of evaluating temporal prediction accuracy. In addition, the validation results indicate that the proposed validation approach has shown great potentials to better evaluate the accuracy of PM2.5 estimation. This study provides application implications and new perspectives for the validation of AOD-PM2.5 models.
BibTeX
@misc{li2018validation,
  author = {Tongwen Li and Huanfeng Shen and Qiangqiang Yuan and Liangpei Zhang},
  title = {{Validation approaches for satellite-based PM2.5 estimation: Assessment and a new approach}},
  year = {2018},
  eprint = {1812.00135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.00135},
}
Figure