Evaluating validation strategies on the performance of soil property prediction from regional to continental spectral data

Songchao Chen, Hanyi Xu, Dongyun Xu, Wenjun Ji, Shuo Li, Meihua Yang, Bifeng Hu, Yin Zhou, Nan Wang, Dominique Arrouays, Zhou ShiZenodo 2021 · DOI

TL;DR

This study investigates how different methods for validating soil property prediction models (using visible-near infrared spectroscopy) perform across various scales, from continental to regional. The key takeaway is that for large datasets, simple validation methods are robust, but for smaller regional datasets, more sophisticated or repeated validation techniques are crucial to avoid misleading results.

What Problem Does This Paper Address?

Predicting soil properties like organic carbon (SOC) and clay content using visible-near infrared (vis-NIR) spectroscopy is a powerful tool for large-scale soil monitoring. However, a critical step is validating these predictive models to ensure their accuracy and reliability. Two common validation approaches are independent validation (splitting data into calibration and testing sets once) and k-fold cross-validation (repeatedly splitting data). The problem is that there's no clear consensus on which validation strategy is best, especially when dealing with data from different geographical scales (continental, national, regional). This lack of standardization makes it difficult to compare results across studies and can lead to over- or under-optimistic assessments of model performance. The authors aim to resolve this by systematically comparing these validation strategies across scales.

Methods

The researchers employed a comprehensive approach to evaluate different validation strategies. They used the Cubist model, a piecewise linear decision tree algorithm, for spectral modeling. The core of their investigation involved comparing two main validation strategies:

  • Independent Validation: This involved splitting the data into calibration and validation sets at different ratios (2:1, 4:1, 9:1). They tested various calibration sampling strategies within this framework:
  • Random Sampling (RS): Selecting samples randomly.
  • Rank: Sorting samples by property value.
  • Kennard-Stone (KS): Selecting samples that are maximally spread in spectral space.
  • Rank-Kennard-Stone (RKS): A combination of rank and KS.
  • Conditioned Latin Hypercube Sampling (cLHS): A stratified sampling method designed to cover both property and spectral spaces.
  • For RS and cLHS, they performed 50 repetitions to assess the variability introduced by random selection.
  • k-fold Cross-Validation: This involved splitting the data into k subsets (3-fold, 5-fold, 10-fold), using one subset for validation and the rest for calibration, and repeating this process for each subset.

To ensure a fair comparison, a consistent test set (20% of total samples) was set aside at each scale and used to evaluate all tested model combinations. Model performance was quantified using Lin's concordance correlation coefficient (CCC).

Datasets

The study utilized the LUCAS (Land Use/Land Cover Area Frame Survey) database, specifically the 2009 topsoil (0-20 cm) samples.

  • Continental Scale: The entire LUCAS dataset of 17,272 mineral soil samples across 25 EU member states.
  • National Scale: A subset of the LUCAS data from France (2761 samples).
  • Regional Scale: Subsets of French data from five administrative regions (Center, Rhône-Alpes, Poitou-Charentes, Lorraine, and Midi-Pyrénées), with sample sizes ranging from 110 to 248.

No new benchmark datasets were introduced; the study leveraged existing, well-established soil spectral data.

Key Insights and Ideas

  • Scale Matters Dramatically: The effectiveness and reliability of validation strategies are highly dependent on the scale of the data. What works well for continental data might be insufficient for regional data.
  • Large Datasets are Forgiving: When you have thousands of samples (continental or national scales), the choice of validation strategy and calibration sampling method has a surprisingly small impact on the final model performance. The models are robust enough to overcome minor variations in validation.
  • Small Datasets are Sensitive: For smaller regional datasets (dozens to hundreds of samples), the validation strategy becomes critically important. A single, random split can lead to highly variable and potentially misleading performance estimates.
  • "No Silver Bullet" for Sampling: At the regional scale, no single calibration sampling strategy consistently outperformed others. This suggests that the complexity of the data and the limited sample size make it hard for any one method to guarantee optimal results.
  • Uncertainty in Single Validation Runs: The high variation observed in CCC values when using a single random sampling or cLHS for calibration on small datasets highlights the inherent uncertainty. This means a single validation run might not accurately reflect the model's true performance.

Key Results

  • Continental & National Scales: Model performance (measured by CCC) for SOC and clay was stable across different calibration sizes, sampling strategies, and validation methods when using large datasets (>1400 samples). Average CCCs for SOC were around 0.83, and for clay, around 0.88-0.89.
  • Regional Scales:
    • A larger calibration size generally improved model performance for smaller datasets (<300 samples).
  • A wider range of soil properties (e.g., wider SOC or clay content) in the calibration set led to better model performance.
  • For small datasets, the variation in CCC among 50 repeated random samplings was substantial (e.g., 0.10-0.44 for SOC), indicating high uncertainty in single validation runs.
  • The study found that for regional datasets, k-fold cross-validation and/or repeated random sampling for independent validation were more robust than a single independent validation run.

Limitations and Future Work

  • Limitations:
    • The study focused on two specific soil properties (SOC and clay) and one modeling algorithm (Cubist). The findings might differ for other properties or models.
  • The "plateau calibration size" (the point where adding more calibration samples doesn't significantly improve performance) can vary based on pedoclimate conditions, soil properties, and models, suggesting a need for more research to establish robust estimates.
  • Future Work:
    • The authors suggest further investigations to confirm their findings regarding the robustness of validation strategies, particularly comparing one-time random sampling with k-fold cross-validation.
  • More research is needed to derive more reliable estimates of the plateau calibration size across different scenarios.
  • The authors call for avoiding single-run random sampling for model evaluation and recommend k-fold cross-validation or repeated random sampling for small datasets.

Summary updated

Abstract

Visible-near infrared (vis–NIR) spectroscopy has been widely used to characterize soil information from field to global scales. Before applying a calibrated spectral predictive model to acquire soil information, either independent validation or <em>k</em>-fold cross validation is used to evaluate model performance. However, there is no consensus on which validation strategy is more suitable and robust when evaluating model performance for the studies in different scales. The objective of this study is to evaluate and compare the model performance of two validation strategies coupling different calibration sizes (a ratio of calibration to validation of 2:1, 4:1 and 9:1) and calibration sampling strategies (random sampling (RS), rank, Kennard-Stone (KS), rank-Kennard-Stone (RKS) and conditioned Latin hypercube sampling (cLHS)) across scales. A total of 17,272 vis–NIR spectra of mineral soils from LUCAS data (continental scale) and their soil organic carbon (SOC) and clay contents were used in this study, and the dataset was further split into national (2761 samples in France) and five regional datasets (110 to 248 samples from five French administrative regions). To eliminate the effect of changing validation set on the model performance, a consistent test set (20% of total samples at each scale) was split to evaluate all the combinations involved in two validation strategies. The Lin’s concordance correlation coefficient (CCC) of the cubist model were stable for both SOC and clay for different calibration sizes, calibration sampling and validation strategies for a large calibration size (&gt;1400) at the national and continental scales. A larger calibration size can potentially improve model performance for a small dataset (&lt;300) at the regional scale, and a wider calibration range would result in better model performance. No silver bullet was found among the different calibration sampling strategies at the regional scale. For five French regions (small data set), we found a high variation (95th percentile minus the 5th percentile) in the CCC among the models built from 50 repeated RS (0.10–0.44 for SOC, 0.16–0.52 for clay) and cLHS (0.08–0.40 for SOC, 0.12–0.36 for clay). This finding indicates that a one-time RS or cLHS for selecting the calibration set has high uncertainty in model evaluation for a small dataset and therefore should be used with caution. Therefore, we suggest the following: (1) for a large data set (thousands), either one-time random sampling for independent validation or <em>k</em>-fold cross validation would be appropriate; (2) for a small data set (dozens to hundreds), <em>k</em>-fold cross validation and/or repeated random sampling for independent validation would be more robust for spectral predictive model evaluation.
BibTeX
@inproceedings{chen2021evaluating,
  author = {Songchao Chen and Hanyi Xu and Dongyun Xu and Wenjun Ji and Shuo Li and Meihua Yang and Bifeng Hu and Yin Zhou and Nan Wang and Dominique Arrouays and Zhou Shi},
  title = {{Evaluating validation strategies on the performance of soil property prediction from regional to continental spectral data}},
  booktitle = {Zenodo},
  year = {2021},
  doi = {10.5281/zenodo.8085605},
}
Figure