Abstract
Before any binary classification model is taken into practice, it is important to validate its performance on a proper test set. Without a frame of reference given by a baseline method, it is impossible to determine if a score is 'good' or 'bad'. The goal of this paper is to examine all baseline methods that are independent of feature values and determine which model is the 'best' and why. By identifying which baseline models are optimal, a crucial selection decision in the evaluation process is simplified. We prove that the recently proposed Dutch Draw baseline is the best input-independent classifier (independent of feature values) for all positional-invariant measures (independent of sequence order) assuming that the samples are randomly shuffled. This means that the Dutch Draw baseline is the optimal baseline under these intuitive requirements and should therefore be used in practice.
Introduction
In the evaluation of binary classification models, performance metrics such as accuracy, precision, and F1-score are standard. However, a high score—say, 90% accuracy—is only meaningful relative to a baseline. If a trivial model that ignores all input features can achieve the same score, the complex machine learning model has failed to provide value. This paper investigates the landscape of "input-independent" baselines and identifies the Dutch Draw as the mathematically optimal choice for evaluating binary classifiers.
The Search for a Robust Baseline
Commonly used baselines, such as predicting the majority class or flipping a weighted coin, are often used arbitrarily. To establish a rigorous benchmark, a baseline must satisfy three core criteria:
- Input-Independence: The model must not utilize feature values ($x$); its predictions are based solely on the number of samples ($M$).
- Positional-Invariance: The performance metric must be independent of the sequence order of the samples. Standard metrics like Accuracy and F1-score meet this requirement.
- Permutation-Optimality: The baseline must demonstrate superior performance on average across all possible shuffles of the test set. This prevents "broken clock" models—deterministic sequences that happen to align with a specific test set by chance—from being considered valid benchmarks.
The Dutch Draw Classifier
The Dutch Draw classifier differs from traditional probabilistic baselines. While a weighted coin flip treats each prediction as an independent random event, the Dutch Draw operates globally. It selects a fixed number of samples ($k$) to be labeled as positive and distributes these labels uniformly at random across the $M$ samples.
The "Dutch Draw baseline" is specifically the version of this classifier where the parameter $k$ is optimized to maximize (or minimize, depending on the goal) the expected performance for a given metric. By treating the test set as a whole rather than a sequence of independent events, the Dutch Draw provides a tighter, more informative performance floor.
Theoretical Insights: The "Broken Clock" and Permutation-Optimality
A significant challenge in baseline selection is the "Broken Clock" problem. A deterministic model that always predicts a specific sequence might achieve a perfect score on one specific test set. However, if the test set is shuffled, that same model would likely fail. The authors argue that a true baseline must be robust against the specific ordering of data.
By requiring permutation-optimality, the authors ensure that the Dutch Draw is not just "lucky." They prove that if one does not account for permutations, the "perfect" input-independent baseline would simply be the ground truth labels themselves—a circular logic that renders the baseline useless. Permutation-optimality provides the statistical rigor necessary to make the Dutch Draw a legitimate floor for performance.
Key Results and Theorem 1
The central contribution of this work is Theorem 1, which formally proves that the optimal Dutch Draw classifier is permutation-optimal among all input-independent classifiers for any positional-invariant measure.
In practical terms, this means:
- Metric Universality: The Dutch Draw is the superior baseline for almost every standard metric used in binary classification.
- Tighter Benchmarking: It provides a more challenging "floor" than a standard weighted coin flip, ensuring that any model beating it has truly learned from the input features.
- Ease of Use: For most common metrics, the Dutch Draw has a closed-form expression, making it computationally simple to implement alongside traditional metrics.
Limitations and Future Directions
While the Dutch Draw establishes a new standard for binary classification, there are areas for further development:
- Multiclass Extension: The current proof is limited to binary outcomes. Extending these mathematical proofs to multiclass scenarios is a logical next step.
- Streaming Data: The Dutch Draw requires knowledge of the total sample size ($M$) to determine the subset $k$. In real-time streaming applications where $M$ is unknown, implementing this baseline is more complex.
- Knowledge of Class Distribution: For certain metrics, the baseline requires knowing the number of positive labels ($P$) in the test set. While this can be estimated from training data, it represents a small dependency on the test set's global properties.
Conclusion
The Dutch Draw represents the optimal input-independent baseline for binary classification. By satisfying the requirements of input-independence, positional-invariance, and permutation-optimality, it provides a mathematically sound and practically useful benchmark. The authors recommend that the Dutch Draw become the standard against which all binary classification models are measured.
Summary updated