An Explorative Study on Estimating Local Accuracies in Land-Cover Information Using Logistic Regression and Class-Heterogeneity-Stratified Data

Jingxiong Zhang, Wenjing Yang, Wangle Zhang, Yu Wang, Di Liu, Yingchang XiuRemote 2018 · DOI

TL;DR

This paper introduces a novel "class-heterogeneity-stratified" approach for estimating local land-cover classification accuracy. By jointly stratifying both map classes and within-class heterogeneity for sampling and modeling, it significantly improves accuracy prediction compared to existing methods, especially in complex landscapes.

What Problem Does This Paper Address?

Traditional land-cover classification accuracy is often reported as a single, overall statistic (e.g., overall accuracy). However, this doesn't tell users where in a map the errors are likely to occur or why. This lack of local accuracy information hinders effective land-cover information refinement and application. Existing methods for estimating local accuracy often rely on empirical models like logistic regression, but they struggle to account for variations in accuracy within a single land-cover class, particularly in heterogeneous areas. This paper aims to address this by proposing a method that specifically handles within-class heterogeneity in both sampling and modeling.

Methods

The core of this research is the class-heterogeneity-stratified method. This approach involves a two-step stratification process:

  1. Class Stratification: Similar to existing methods, the data is first divided into strata based on the land-cover map classes (e.g., forest, water, cultivated land).
  2. Heterogeneity Stratification: Within each class stratum, pixels are further divided into sub-strata based on their local heterogeneity. Heterogeneity is quantified by looking at the diversity of land-cover classes within a small neighborhood (e.g., a 3x3 pixel window) around a given pixel. Pixels in neighborhoods with many different classes are considered "heterogeneous," while those in neighborhoods with mostly the same class are "homogeneous."

This double-stratification is applied to both:

  • Sampling Design: More sample data is strategically allocated to the heterogeneous sub-strata, as these are more prone to misclassification.
  • Logistic Regression Modeling: Separate logistic regression models are built for each homogeneous and heterogeneous sub-stratum within each class. This allows the models to capture the unique accuracy-context relationships present in each specific sub-stratum.

The paper compares this novel method against simpler approaches: class-aggregated (CA) and class-stratified (CS) modeling, and a simpler double-stratification method (EI) that distinguishes between edge and interior pixels.

Datasets

  • Training and Testing Data: The study utilizes the GlobeLand30 2010 land-cover dataset for Wuhan, China.
  • Reference Data: High-resolution Google Earth imagery was used for visual interpretation to create reference class labels for the sample pixels. When Google Earth imagery was unavailable, ground visits and Landsat TM imagery were used.

Key Insights and Ideas

  • Heterogeneity Matters: The research strongly reinforces that landscape heterogeneity is a critical factor influencing classification accuracy. Areas with a mix of land-cover types are inherently harder to classify accurately.
  • Joint Sampling and Modeling is Crucial: The paper highlights that effectively addressing local accuracy requires a tightly coupled approach to both sampling and modeling. Simply stratifying for one without the other limits the potential gains.
  • Double Stratification for Accuracy: The proposed class-heterogeneity stratification, applied to both sampling and modeling, proved to be the most effective way to capture and predict local accuracy variations.
  • Sub-strata Models are Key: Building separate models for homogeneous and heterogeneous sub-strata within each class is essential for accurately reflecting within-class accuracy variations.

Key Results

  • Superior Accuracy: The proposed class-heterogeneity-stratified method (EO) consistently achieved the highest Area Under the Curve (AUC) values for local accuracy estimation, indicating superior predictive performance.
  • Significant Improvements: EO demonstrated statistically significant improvements in AUC over simpler methods like CA and CS, especially when using larger sample sizes.
  • Robustness to Sample Size: The EO method showed good robustness to varying sample sizes, maintaining its competitive performance even with reduced datasets, suggesting cost-effectiveness.
  • Class-Specific Models: The study found that optimal models varied significantly across different land-cover classes and sub-strata, underscoring the need for tailored modeling.

Limitations and Future Work

  • Two-Level Uncertainty: The authors acknowledge a significant limitation: the explanatory variables (contextual features) are derived from the land-cover map itself, which is already subject to misclassification errors. This creates a "two-level uncertainty" that propagates into the local accuracy estimates. Future work should focus on methods to handle and incorporate this uncertainty.
  • Kriging for Spatial Correlation: The study did not explore logistic-regression-kriging, which could potentially improve accuracy by accounting for spatial correlation in regression residuals. This is suggested as a future research direction.
  • Simplification of Sub-Stratified Models: The authors suggest exploring ways to simplify the complex sub-stratified models without significantly compromising accuracy, perhaps by using sub-stratum means.
  • Temporal Extension: Investigating the application of the double-stratification method for characterizing per-pixel accuracies in land-cover change scenarios is proposed.
  • Adaptive Threshold Selection: The paper highlights the importance of adaptively selecting homogeneity thresholds for sub-stratification, which could optimize accuracy estimation under varying landscape conditions.
  • Fuzzy Classifications: The current method is designed for crisp classifications. Extending it to fuzzy or subpixel classifications would be a valuable future direction.

Summary updated

Abstract

It is increasingly recognized that classification accuracy should be characterized locally at the level of individual pixels to depict its spatial variability to better inform users and producers of land-cover information than by conventional error-matrix-based methods. Local or per-pixel accuracy is usually estimated through empirical modelling, such as logistic regression, which often proceeds in a class-aggregated or a class-stratified way, with the latter being generally more accurate due to its accommodation for between-class inhomogeneity in accuracy-context relations. As an extension to class-stratified modelling, class-heterogeneity-stratified modelling, in which logistic models are built separately for contextually heterogeneous vs. homogeneous sub-strata in individual strata of map classes, is proposed in this paper for proper handling of within-class inhomogeneity in accuracy-context relations to increase accuracy of estimation. Unlike in existing literature where sampling is usually approached separately, the double-stratification method is also adopted in sampling design so that more sample data are likely allocated to heterogeneous sub-strata (which are more prone to misclassifications than homogeneous ones). This class-heterogeneity-stratified method furnished for sampling and modelling jointly thus constitutes an integrative framework for accuracy estimation and information refinement. As the first step in building up such a framework, this paper investigates the proposed double-stratification method’s performance and sensitivity to sample size regarding local accuracy estimation in comparison with those of existing methods through a case study concerning Globeland30 2010 land cover over Wuhan, China. A detailed review of existing methods for analyses, estimation, and use of local accuracy was provided, helping to put the proposed research in a broader context. Candidate explanatory variables for logistic regression included sample pixels’ map classes, positions, and contextual features that were computed in different-sized moving windows. Relative performances of these methods were evaluated based on an independent reference sample, with all methods found reliable. It was confirmed that the proposed method is in general the most accurate, as observed with varying sample sizes. The proposed method’s competitive performance is thus proved, reinforcing its potential for information refinement. Extensions to and uncertainty aspects of the proposed method were discussed, with further research proposed.
BibTeX
@inproceedings{zhang2018explorative,
  author = {Jingxiong Zhang and Wenjing Yang and Wangle Zhang and Yu Wang and Di Liu and Yingchang Xiu},
  title = {{An Explorative Study on Estimating Local Accuracies in Land-Cover Information Using Logistic Regression and Class-Heterogeneity-Stratified Data}},
  booktitle = {Remote},
  year = {2018},
  doi = {10.3390/rs10101581},
}
Figure