Accuracy Assessment of GlobeLand30 2010 Land Cover over China Based on Geographically and Categorically Stratified Validation Sample Data

Yu Wang, Jingxiong Zhang, Di Liu, Wenjing Yang, Wangle ZhangRemote 2018 · DOI

TL;DR

This paper presents the first statistically rigorous, country-wide accuracy assessment of the GlobeLand30 2010 land cover dataset for China. Using a novel two-level stratified sampling approach and visual interpretation of Google Earth imagery, they found an overall accuracy of 84.2%, indicating GlobeLand30's high utility for many applications, while also highlighting areas for improvement in mapping complex landscapes.

What Problem Does This Paper Address?

Accurate land cover information is crucial for understanding Earth's natural resources and environmental processes. While global land cover products like GlobeLand30 offer valuable 30m resolution data, their reliability needs thorough validation, especially at a national scale. Previous assessments for China were either not statistically robust, used coarser resolutions, or lacked comprehensive coverage. This paper aims to fill this gap by providing a statistically sound, geographically stratified accuracy assessment of GlobeLand30 2010 over China, enabling users to better understand its strengths and limitations.

Methods

The authors employed a rigorous two-level stratified random sampling design. The first level stratified China into ten geographical regions, and the second level stratified by GlobeLand30's land cover classes. A total of 8400 validation sample pixels were collected. Reference data was acquired through visual interpretation of high-resolution Google Earth imagery by experienced interpreters. To account for potential ambiguities, a "less strict" agreement definition (matching map label with either primary or alternate reference label) was used for the main analysis, yielding an overall accuracy of 84.2%. A "stricter" definition (matching map label only with the primary reference label) was also used for comparison, resulting in a lower overall accuracy of 81.0%. Error matrices were constructed for each region and then aggregated using appropriate weighting to derive national accuracy estimates.

Datasets

  • GlobeLand30 2010: The primary land cover dataset being assessed.
  • Google Earth High-Resolution Satellite Imagery: Used as the primary source for visual interpretation to create reference land cover labels for validation pixels.
  • Other available datasets (Bing maps, Yahoo maps, color composites of Landsat images): Utilized when Google Earth historical imagery was not of sufficient quality.

The paper does not introduce new benchmark datasets but rather establishes a robust validation methodology for an existing one.

Key Insights and Ideas

  • The importance of stratified sampling: The two-level stratification (geographic and thematic) is crucial for obtaining statistically sound and regionally representative accuracy estimates, allowing for both national and regional reporting.
  • Defining "agreement" matters: The choice of agreement definition (primary vs. primary/alternate reference label) significantly impacts accuracy estimates. The "less strict" definition, which accounts for potential ambiguities in visual interpretation, provides a more realistic assessment of the product's utility for many applications.
  • Spatial heterogeneity drives accuracy variations: Accuracy is not uniform across China. Regions with more complex and fragmented landscapes, or where certain land cover classes are rare, tend to exhibit lower accuracies.
  • Class-specific challenges: Certain land cover classes, like shrubland, wetland, and artificial surfaces, are more prone to misclassification, highlighting the need for improved classification algorithms or protocols for these categories.

Key Results

  • National Overall Accuracy: 84.2% (using the less strict agreement definition).
  • National Overall Accuracy (stricter definition): 81.0%.
  • High accuracies for dominant classes: User's and producer's accuracies were generally above 80% for cultivated land, forest, permanent snow and ice, and bareland.
  • Specific class accuracies: User's accuracy for water bodies was high at 94.2%, while wetland and artificial surface accuracies were lower (82.1% and 79.8%, respectively).
  • Regional variations: Overall accuracies ranged from 76.0% to 90.3%, with northern regions generally showing higher accuracies than southern regions.
  • Misclassification patterns: Dominant classes were sometimes overestimated, and spectrally similar classes (grassland, shrubland, forest) were difficult to distinguish. Fragmented patches and "salt-and-pepper" noise were also identified as sources of error.

Limitations and Future Work

  • Limitations:
    • The "less strict" agreement definition, while practical, might mask some finer classification errors.
  • The study focused on 2010 data; land cover is dynamic, and accuracy may change over time.
  • The visual interpretation process, while rigorous, is still subject to interpreter bias and the quality of available imagery.
  • The paper notes that areas with heterogeneous landscapes and scattered small patches require improved mapping methods.
  • Future Work:
    • The authors suggest that future global land cover mapping at fine spatial resolution needs to address challenges in mapping scattered classes of small extents and spectrally mixed pixels.
  • Further research is needed to develop and refine mapping protocols to better distinguish classes prone to misclassification, such as grassland, shrubland, and artificial surfaces.
  • The GlobeLand30 production and validation teams can benefit from these findings to improve future iterations of the dataset.

Summary updated

Abstract

Land cover information is vital for research and applications concerning natural resources and environmental modeling. Accuracy assessment is an important dimension in use and production of land cover information. GlobeLand30 is a relatively new global land cover information product with a fine spatial resolution of 30 m and is potentially useful for many applications. This paper describes the methods for and results from the first country-wide and statistically based accuracy assessment of GlobeLand30 2010 land cover dataset over China. For this, a total of 8400 validation sample pixels were collected based on a sampling design featuring two levels of stratification (ten geographical regions, each with nine or eight land-cover classes). Validation sample data with reference class labels were acquired from visual interpretation based on Google Earth high-resolution satellite images. Error matrices for individual regions and entire China were estimated properly based on the sampling design adopted, with the former aggregated to get the latter through suitable weighting. Results were obtained, with agreement at a sample pixel defined both as a match between the map (class) label and either the primary or alternate reference label therein and, more strictly, as a match between the map label and the primary reference label only. Based on the former definition of agreement, the overall accuracy of GlobeLand30 2010 land cover for China was assessed to be 84.2%. User’s accuracy and producer’s accuracy were both greater than 80% for cultivated land, forest, permanent snow and ice, and bareland, with user’s accuracy for water bodies estimated 94.2% (82.1% for wetland, 79.8% for artificial surface) and producer’s accuracy for grassland estimated 89.0%. These indicate that GlobeLand30 2010 depicts land cover circa 2010 in China quite accurately, although estimates of accuracy indicators based on the latter definition of agreement were lower as expected with an estimated national overall accuracy of 81.0%. Regional and class variations in accuracy were revealed and examined in the light of their associations with land cover distributions and patterns. Implications for use and production of GlobeLand30 land cover information were discussed, so were commonality and lack of it between GlobeLand30 and other fine-resolution land cover products.
BibTeX
@inproceedings{wang2018accuracy,
  author = {Yu Wang and Jingxiong Zhang and Di Liu and Wenjing Yang and Wangle Zhang},
  title = {{Accuracy Assessment of GlobeLand30 2010 Land Cover over China Based on Geographically and Categorically Stratified Validation Sample Data}},
  booktitle = {Remote},
  year = {2018},
  doi = {10.3390/rs10081213},
}
Figure