Abstract
K-fold cross validation (CV) is a popular method for estimating the true performance of machine learning models, allowing model selection and parameter tuning. However, the very process of CV requires random partitioning of the data and so our performance estimates are in fact stochastic, with variability that can be substantial for natural language processing tasks. We demonstrate that these unstable estimates cannot be relied upon for effective parameter tuning. The resulting tuned parameters are highly sensitive to how our data is partitioned, meaning that we often select sub-optimal parameter choices and have serious reproducibility issues. Instead, we propose to use the less variable J-K-fold CV, in which J independent K-fold cross validations are used to assess performance. Our main contributions are extending J-K-fold CV from performance estimation to parameter tuning and investigating how to choose J and K. We argue that variability is more important than bias for effective tuning and so advocate lower choices of K than are typically seen in the NLP literature and instead use the saved computation to increase J. To demonstrate the generality of our recommendations we investigate a wide range of case-studies: sentiment classification (both general and target-specific), part-of-speech tagging and document classification.
Introduction: The Problem of Internal Variability
In the quest for state-of-the-art performance, NLP researchers often celebrate accuracy gains of less than 1%. However, the very tool used to measure these gains—$K$-fold cross-validation (CV)—is often noisier than the improvements being measured.
The "dirty secret" of CV is internal variability: the fact that your performance estimate changes depending on how you randomly shuffle and partition your data into folds. If this variability is larger than the performance difference between two sets of hyperparameters, your tuning process is essentially a coin flip. This leads to "sub-optimal" models and a massive reproducibility crisis where other researchers cannot replicate your "best" settings because they used a different random seed for their data splits.
Methods: Moving to J-K-fold Cross Validation
The authors propose a shift from standard $K$-fold CV to $J$-$K$-fold Cross Validation (also known as repeated cross-validation) specifically for parameter tuning.
- The Approach: Instead of running one round of $K$-fold CV, you run $J$ independent rounds, each with a different random partitioning of the data. You then average the results across all $J \times K$ evaluations.
- The Strategy: The authors challenge the industry standard of 10-fold CV. They argue that for tuning, variance is a bigger enemy than bias. They recommend using a lower $K$ (e.g., $K=5$) and using the saved computational budget to increase the number of repetitions ($J$).
- Tuning: They integrate this into a standard grid search, showing that the averaged estimates provide a much smoother and more reliable "surface" for finding the best parameters.
Key Insights
- Bias vs. Variance in Tuning: In performance estimation, we care about bias (how close we are to the "true" accuracy). In tuning, we only care about the relative ranking of parameters. As long as the bias is consistent across the grid, high variance is what actually prevents us from finding the peak.
- The "Rule of 10" is a Myth: Many researchers use 10-fold CV because it's a common default. The paper shows that 2-5-fold CV (two repetitions of 5-fold) is often more stable than 1-10-fold CV for the exact same computational cost.
- NLP Sensitivity: Unlike simple tabular data, language has complex structural properties (discourse, syntax). Simply balancing class proportions (stratification) isn't enough to ensure a "representative" split, making repeated CV even more critical for NLP than for other fields.
Key Results
The authors validated their approach across four diverse NLP tasks: IMDB Movie Reviews, Brown Corpus POS tagging, Reuters-21578 document classification, and Twitter target-dependent sentiment analysis.
- Reduced Uncertainty: On the IMDB dataset, moving from a single 5-fold CV to a 10-5-fold CV (10 repetitions) cut the standard deviation of the chosen hyperparameter nearly in half (from 0.0427 to 0.0221).
- Stability in Deep Learning: For LSTMs, a single 10-fold CV was found to be "not at all suitable" for tuning, producing almost random results for hidden layer width. Using 8-5-fold CV produced a clear, consistent winner.
- Visualizing the Noise: The authors used histograms of "chosen parameters" across 1,000 different random seeds. While standard CV produced a wide, flat distribution of "best" parameters, $J$-$K$-fold CV consistently converged on a single, dominant bin.
Limitations and Future Work
$J$-$K$-fold CV is $J$ times more expensive than a single round. While the authors suggest lowering $K$ to compensate, high-resource models (like massive Transformers) may still find this prohibitive. Future work includes developing adaptive repetitions—stopping once variance drops below a threshold—and better integration with early stopping in neural network training.
Summary updated