Abstract
We present FIESTA, a model selection approach that significantly reduces the computational resources required to reliably identify state-of-the-art performance from large collections of candidate models. Despite being known to produce unreliable comparisons, it is still common practice to compare model evaluations based on single choices of random seeds. We show that reliable model selection also requires evaluations based on multiple train-test splits (contrary to common practice in many shared tasks). Using bandit theory from the statistics literature, we are able to adaptively determine appropriate numbers of data splits and random seeds used to evaluate each model, focusing computational resources on the evaluation of promising models whilst avoiding wasting evaluations on models with lower performance. Furthermore, our user-friendly Python implementation produces confidence guarantees of correctly selecting the optimal model. We evaluate our algorithms by selecting between 8 target-dependent sentiment analysis methods using dramatically fewer model evaluations than current model selection approaches.
1. Introduction: The Noise in Model Selection
In modern Natural Language Processing (NLP), claiming "State-of-the-Art" (SOTA) status is the gold standard. However, evaluating models is surprisingly "noisy." A model’s performance can swing significantly based on two primary factors: data splits (which specific rows end up in the training vs. testing set) and random seeds (stochastic elements like weight initialization, dropout, and data shuffling).
Common practice involves comparing models based on a single run (one seed, one split), which is statistically unreliable. Conversely, exhaustive methods like K-fold cross-validation or running dozens of random seeds are prohibitively expensive for large, modern models. FIESTA addresses this "efficiency vs. reliability" trade-off, providing a way to be certain about the winner without breaking the compute budget.
As illustrated in the figure above, performance distributions can overlap significantly. A model might appear superior on one specific split but fail to maintain that lead across others, making adaptive evaluation essential for robust selection.
2. The FIESTA Framework
FIESTA frames model selection as a Best-Arm Identification problem from bandit theory. Instead of pulling a slot machine arm to get a jackpot, a researcher "pulls" a model by training and evaluating it on a random seed/split to obtain a performance score.
The framework introduces three specific algorithms for different research scenarios:
- Sequential Halving (SH): Best for a Fixed Budget. If total model runs are limited, SH divides them into rounds. After each round, it discards the bottom 50% of models and reallocates the remaining budget to the "survivors."
- Top-Two Thompson Sampling (TTTS): Best for Fixed Confidence. If a researcher requires a specific confidence level (e.g., 95%), TTTS adaptively chooses which model to run next by comparing the current leader against its most likely challenger.
- Batch Thompson Sampling (BTS): An extension of TTTS designed for parallel computing, allowing researchers to evaluate multiple models or seeds simultaneously across a cluster.
3. Experimental Evaluation
The researchers evaluated FIESTA using the SemEval 2014 Task 4 (Restaurant Dataset), a standard benchmark for Target-Dependent Sentiment Analysis (TDSA). They compared 8 to 12 different TDSA model architectures (including variants of ATAE, IAN, and TDLSTM) to simulate a real-world model selection challenge.
4. Results and Discussion
FIESTA demonstrates significant efficiency gains and improved reliability over standard non-adaptive baselines. In the TDSA experiments, Top-Two Thompson Sampling (TTTS) reached the required confidence level using roughly half the total evaluations compared to a non-adaptive baseline.
Under a fixed budget of 204 evaluations, Sequential Halving correctly identified the best model 99% of the time, whereas the standard non-adaptive approach was only correct 85% of the time. By stopping evaluations for clearly inferior models early, FIESTA focuses computational resources where they matter most: the "photo finish" between the top contenders. In these tasks, FIESTA consistently identified TDLSTM as the SOTA model, even when performance differences between the top three models were very slim.
5. Conclusion
FIESTA offers a much-needed statistical sanity check for the NLP community. It proves that rigorous comparisons and computational efficiency are not mutually exclusive—researchers simply need to be smarter about which "arms" they pull. While the current implementation assumes Gaussian performance distributions, it provides a robust framework for identifying SOTA models with high confidence. Future work will explore more robust "heavy-tailed" distributions for noise and develop even more aggressive batching strategies for massive-scale parallelization.
Summary updated