F1-Optimal Thresholding in the Multi-Label Setting

Zachary Chase Lipton, Charles Elkan, Balakrishnan NarayanaswamyarXiv 2014 · OpenReview

Abstract

This paper provides new insight into maximizing F1 scores in the context of binary classification and also in the context of multilabel classification. The harmonic mean of precision and recall, F1 score is widely used to measure the success of a binary classifier when one class is rare. Micro average, macro average, and per instance average F1 scores are used in multilabel classification. For any classifier that produces a real-valued output, we derive the relationship between the best achievable F1 score and the decision-making threshold that achieves this optimum. As a special case, if the classifier outputs are well-calibrated conditional probabilities, then the optimal threshold is half the optimal F1 score. As another special case, if the classifier is completely uninformative, then the optimal behavior is to classify all examples as positive. Since the actual prevalence of positive examples typically is low, this behavior can be considered undesirable. As a case study, we discuss the results, which can be surprising, of applying this procedure when predicting 26,853 labels for Medline documents.

Introduction

The F1 score—the harmonic mean of precision and recall—is the standard metric for binary classification on imbalanced data. Unlike accuracy, where a 0.5 threshold is often optimal for calibrated probabilities, the optimal threshold for F1 is non-linear and depends on both the data distribution and the classifier's performance. In multi-label classification, the challenge scales with the choice of metric: Micro-F1 (global), Macro-F1 (label-average), or Per-instance F1. This paper explores how these metrics fundamentally change thresholding strategies and exposes a "pathological" behavior where Macro-F1 can incentivize predicting rare labels for every instance.

Theoretical Framework

The authors utilize "plug-in rules" to optimize F1 scores, finding the best threshold for pre-trained model scores. A central contribution is Theorem 1, which derives the optimal threshold for any real-valued classifier.

The Half-F1 Rule

For classifiers outputting well-calibrated conditional probabilities, a key corollary emerges: the optimal threshold $t$ is exactly half of the maximum achievable F1 score ($F^*/2$). Because F1 cannot exceed 1.0, the optimal threshold for a calibrated classifier can never exceed 0.5.

F1 score behavior relative to true positives at a base rate of 0.1.

Batch Dependency

A critical insight is the batch dependency of F1 optimization. The optimal threshold for a single instance depends on the scores of all other instances in the batch. One cannot determine if a specific probability (e.g., 0.4) is "positive" in isolation; the entire distribution of scores is required to estimate the potential F1.

The Macro-F1 Trap and the Uninformative Case

Macro-F1 weights every label equally, regardless of frequency. For rare labels where a model is completely uninformative (assigning the same score to everything), Theorem 3 proves that the strategy maximizing expected F1 is to predict all examples as positive.

Expected F1 score as a function of the percentage of instances predicted positive, illustrating the incentive to over-predict in uninformative cases.

While the resulting F1 score for that label is low, it is higher than the zero F1 achieved by predicting all negatives. In a Macro-average across thousands of labels, these "junk" gains on rare categories can significantly inflate the total score, leading to massive over-prediction.

Empirical Validation: Medline/MeSH Case Study

The theory was tested on the Medline/MeSH dataset, involving 26,853 distinct labels. Using a linear regression model, the authors observed the predicted pathology. When optimizing for Macro-F1, the system predicted extremely rare terms for more than one-third of all test documents. For example, the term "Platypus" was assigned to thousands of documents where it did not belong, simply because the classifier was uninformative for that label and "predict all" was the F1-optimal strategy.

Conclusion and Limitations

While the $t = F^*/2$ rule provides an elegant solution for calibrated models, practical application faces challenges. Setting thresholds on small validation sets can lead to "the winner’s curse," causing overfitting and suboptimal performance on new data. Furthermore, the reliance on calibration means that biased model probabilities will degrade the rule's effectiveness. These findings suggest a need for caution when using Macro-F1 as an optimization target in large-scale multi-label settings.

Summary updated

Abstract

This paper provides new insight into maximizing F1 scores in the context of binary classification and also in the context of multilabel classification. The harmonic mean of precision and recall, F1 score is widely used to measure the success of a binary classifier when one class is rare. Micro average, macro average, and per instance average F1 scores are used in multilabel classification. For any classifier that produces a real-valued output, we derive the relationship between the best achievable F1 score and the decision-making threshold that achieves this optimum. As a special case, if the classifier outputs are well-calibrated conditional probabilities, then the optimal threshold is half the optimal F1 score. As another special case, if the classifier is completely uninformative, then the optimal behavior is to classify all examples as positive. Since the actual prevalence of positive examples typically is low, this behavior can be considered undesirable. As a case study, we discuss the results, which can be surprising, of applying this procedure when predicting 26,853 labels for Medline documents.
BibTeX
@misc{lipton2014f,
  author = {Zachary Chase Lipton and Charles Elkan and Balakrishnan Narayanaswamy},
  title = {{F1-Optimal Thresholding in the Multi-Label Setting}},
  year = {2014},
  eprint = {1402.1892},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1402.1892},
}
Figure