Abstract
This study investigates the inability of two popular data splitting techniques—train/test split and k-fold cross-validation—to create training and validation datasets that achieve sufficient generality for supervised deep learning (DL) methods. This failure is primarily caused by their limited ability to generate new data views. In response, the bootstrap is a computer-based statistical resampling method used efficiently for estimating the distribution of a sample estimator and assessing a model without knowledge of the population. This paper couples cross-validation and bootstrap to leverage their respective advantages in data generation, achieving better generalization of a DL model. The proposed method, kCV-B, is applied to semantic segmentation and demonstrated via the PointNet algorithm using aerial laser scanning point cloud data.
Introduction
Deep learning (DL) models are notoriously data-hungry and prone to overfitting. A major challenge in developing these models is ensuring they generalize well to unseen data. Standard practices like a simple train/test split or k-fold cross-validation (kCV) often fall short because they rely on a single, static view of the available data. These methods may fail to represent the true underlying population; if a validation set happens to be a non-representative subset, the model may appear performant during training but fail in real-world applications. Furthermore, there is often a lack of statistical rigor in assessing the quality of performance estimators like Mean Square Error (MSE).
Methods: The kCV-B Algorithm
The authors propose kCV-B (Bootstrap with Cross-Validation), a hybrid resampling strategy designed to improve model selection and provide statistically grounded performance metrics.
The algorithm follows a novel four-step sequence:
- Coupling kCV and Bootstrap: Data is split into $k$ folds. For each fold, a model is trained on $k-1$ folds. Instead of validating on the remaining fold once, $B$ independent bootstrap samples (resampling with replacement) are drawn from that validation fold.
- Model Development: Using architectures like PointNet, $k$ different models are developed based on the training sets.
- Statistical Inference: For each model, the MSE is calculated across all $B$ bootstrap validation sets. This allows for the calculation of the Mean MSE ($M_{MSE}$), Standard Error ($SE_{MSE}$), and 95% Confidence Intervals.
- Model Selection & Assessment: The "best" model is selected based on the lowest $M_{MSE}$ and $SE_{MSE}$. The final model is then retrained using the optimal training/validation pair identified by this process.
Results and Discussion
The method was validated using two large-scale aerial LiDAR datasets: DALES (Surrey, Canada) and AHN3 (Netherlands).
Key Insights
- Data "Creation" via Resampling: While bootstrapping doesn't create new physical data points, it creates new combinations. This "multiplicity" of validation sets acts as a better proxy for real-world variety.
- Robustness to Imbalance: kCV-B handled rare classes exceptionally well. In the DALES dataset, kCV-B identified "building" points with an F1-score of 83.6%, significantly higher than the 68.2% managed by standard kCV. It was also more effective at detecting thin structures like power lines.
- Statistical Safety Net: By using confidence intervals for MSE (as seen in the figure above), the researchers move away from point estimates toward a more honest understanding of model fluctuation.
Performance Metrics
kCV-B consistently outperformed traditional methods:
- DALES Dataset: kCV-B achieved a Mean Overall Accuracy ($M_{OA}$) of 83.6%, compared to 81% for kCV and 79.4% for a standard train/test split.
- AHN3 Dataset: On a held-out test set, kCV-B reached 91.1% accuracy. In contrast, a standard train/test split showed 95.3% accuracy on its internal validation set but crashed to 81.6% on the actual test data—a clear sign of overfitting that kCV-B successfully avoided.
Conclusion
The primary drawback of kCV-B is its computational cost, as training $k$ models and evaluating them against $k \times B$ validation sets is time-consuming. However, the gain in generalization and statistical reliability is substantial. Future work will focus on developing more efficient bootstrap approaches and testing the method on even larger, more diverse datasets.
Summary updated