kCV-B: BOOTSTRAP WITH CROSS-VALIDATION FOR DEEP LEARNING MODEL DEVELOPMENT, ASSESSMENT AND SELECTION

Abdul Nurunnabi, Felix Norman Teferle, Debra F. Laefer, Fabio Remondino, İsmail Rakıp Karaş, Jonathan Li2022 · DOI

Abstract

This study investigates the inability of two popular data splitting techniques—train/test split and k-fold cross-validation—to create training and validation datasets that achieve sufficient generality for supervised deep learning (DL) methods. This failure is primarily caused by their limited ability to generate new data views. In response, the bootstrap is a computer-based statistical resampling method used efficiently for estimating the distribution of a sample estimator and assessing a model without knowledge of the population. This paper couples cross-validation and bootstrap to leverage their respective advantages in data generation, achieving better generalization of a DL model. The proposed method, kCV-B, is applied to semantic segmentation and demonstrated via the PointNet algorithm using aerial laser scanning point cloud data.

Introduction

Deep learning (DL) models are notoriously data-hungry and prone to overfitting. A major challenge in developing these models is ensuring they generalize well to unseen data. Standard practices like a simple train/test split or k-fold cross-validation (kCV) often fall short because they rely on a single, static view of the available data. These methods may fail to represent the true underlying population; if a validation set happens to be a non-representative subset, the model may appear performant during training but fail in real-world applications. Furthermore, there is often a lack of statistical rigor in assessing the quality of performance estimators like Mean Square Error (MSE).

Methods: The kCV-B Algorithm

The authors propose kCV-B (Bootstrap with Cross-Validation), a hybrid resampling strategy designed to improve model selection and provide statistically grounded performance metrics.

Schematic of the kCV-B data splitting and bootstrap resampling process.

The algorithm follows a novel four-step sequence:

  1. Coupling kCV and Bootstrap: Data is split into $k$ folds. For each fold, a model is trained on $k-1$ folds. Instead of validating on the remaining fold once, $B$ independent bootstrap samples (resampling with replacement) are drawn from that validation fold.
  2. Model Development: Using architectures like PointNet, $k$ different models are developed based on the training sets.
  3. Statistical Inference: For each model, the MSE is calculated across all $B$ bootstrap validation sets. This allows for the calculation of the Mean MSE ($M_{MSE}$), Standard Error ($SE_{MSE}$), and 95% Confidence Intervals.
  4. Model Selection & Assessment: The "best" model is selected based on the lowest $M_{MSE}$ and $SE_{MSE}$. The final model is then retrained using the optimal training/validation pair identified by this process.

Results and Discussion

The method was validated using two large-scale aerial LiDAR datasets: DALES (Surrey, Canada) and AHN3 (Netherlands).

Statistical inference of model performance showing MSE distributions and confidence intervals.

Key Insights

  • Data "Creation" via Resampling: While bootstrapping doesn't create new physical data points, it creates new combinations. This "multiplicity" of validation sets acts as a better proxy for real-world variety.
  • Robustness to Imbalance: kCV-B handled rare classes exceptionally well. In the DALES dataset, kCV-B identified "building" points with an F1-score of 83.6%, significantly higher than the 68.2% managed by standard kCV. It was also more effective at detecting thin structures like power lines.
  • Statistical Safety Net: By using confidence intervals for MSE (as seen in the figure above), the researchers move away from point estimates toward a more honest understanding of model fluctuation.

Performance Metrics

kCV-B consistently outperformed traditional methods:

  • DALES Dataset: kCV-B achieved a Mean Overall Accuracy ($M_{OA}$) of 83.6%, compared to 81% for kCV and 79.4% for a standard train/test split.
  • AHN3 Dataset: On a held-out test set, kCV-B reached 91.1% accuracy. In contrast, a standard train/test split showed 95.3% accuracy on its internal validation set but crashed to 81.6% on the actual test data—a clear sign of overfitting that kCV-B successfully avoided.

Conclusion

The primary drawback of kCV-B is its computational cost, as training $k$ models and evaluating them against $k \times B$ validation sets is time-consuming. However, the gain in generalization and statistical reliability is substantial. Future work will focus on developing more efficient bootstrap approaches and testing the method on even larger, more diverse datasets.

Summary updated

Abstract

This study investigates the inability of two popular data splitting techniques: train/test split and k-fold cross-validation that are to create training and validation data sets, and to achieve sufficient generality for supervised deep learning (DL) methods. This failure is mainly caused by their limited ability of new data creation. In response, the bootstrap is a computer based statistical resampling method that has been used efficiently for estimating the distribution of a sample estimator and to assess a model without having knowledge about the population. This paper couples cross-validation and bootstrap to have their respective advantages in view of data generation strategy and to achieve better generalization of a DL model. This paper contributes by: (i) developing an algorithm for better selection of training and validation data sets, (ii) exploring the potential of bootstrap for drawing statistical inference on the necessary performance metrics (e.g., mean square error), and (iii) introducing a method that can assess and improve the efficiency of a DL model. The proposed method is applied for semantic segmentation and is demonstrated via a DL based classification algorithm, PointNet, through aerial laser scanning point cloud data.
BibTeX
@inproceedings{nurunnabi2022i,
  author = {Abdul Nurunnabi and Felix Norman Teferle and Debra F. Laefer and Fabio Remondino and İsmail Rakıp Karaş and Jonathan Li},
  title = {{<i>k</i>CV-B: BOOTSTRAP WITH CROSS-VALIDATION FOR DEEP LEARNING MODEL DEVELOPMENT, ASSESSMENT AND SELECTION}},
  booktitle = {˜The},
  year = {2022},
  doi = {10.5194/isprs-archives-xlviii-4-w3-2022-111-2022},
}
Figure