When Disagreement Labels Do Not Identify F1 Rankings

Roberto Fernández-Barrios, Iker Pastor-López, Asier González-Santocildes, Pablo García BringasarXiv 2026 · arXiv

TL;DR

When comparing two machine learning models using the F1 score, simply labeling the items where they disagree isn't enough to guarantee which model is better. This paper introduces a formal "audit" method that uses disagreement labels and a minimal number of agreement queries to certify model rankings within a specific tolerance, cutting labeling costs by more than half in some scenarios.

What Problem Does This Paper Address?

In model evaluation, we often try to save time and money by only labeling "discordant pairs"—the instances where two models give different predictions. For metrics like accuracy, this works perfectly because the instances where models agree cancel each other out in the comparison.

However, the authors point out a critical trap: Agreement labels do not cancel out for F1 rankings. Because the F1 score is a ratio that depends on the total number of true positives, the ground truth of items where both models agree (e.g., both predicted "Positive") can flip the ranking of which model is superior.

The paper addresses the challenge of providing a deterministic guarantee for F1 rankings on a specific dataset batch without requiring prior knowledge of the data's prevalence or the models' baseline performance.

Methods

The authors develop an explicit audit criterion based on partial identification. They break down the unknown labels into two counts: positive labels among (0,0) agreements and positive labels among (1,1) agreements.

Key components include:

  • Sharp Sign Criterion: A mathematical formula that determines if a "winner" is guaranteed regardless of what the remaining unknown labels are.
  • Exact Magnitude Bounds: An $O(n)$ algorithm to calculate the interval $[L, U]$ representing the minimum and maximum possible difference in F1 scores between two models.
  • Bounded Regret Choice: A decision rule that allows an auditor to stop labeling once the potential "loss" (regret) of picking the wrong model falls below a user-defined tolerance ($\epsilon$).
  • Labeling Policies: They compare different strategies for which agreement items to label first, including "Magnitude" ordering, which prioritizes queries that most quickly shrink the uncertainty in the F1 difference.

Datasets

The study performs a retrospective evaluation using 3,200 model pairs across 36 different scenarios. They utilized the TDC (Target-Domain Certificates) prediction artifact, which includes:

  • TableShift: BRFSS diabetes and ACS FoodStamps datasets.
  • Security Datasets: EMBER (malware detection), ToN-IoT, and UNSW-NB15 (network intrusion).
  • Models: Each scenario included a quantum-kernel predictor and 115 classical model configurations, evaluated on batches of 500 items.

Key Insights and Ideas

  • The F1 Ambiguity: Even if you know the exact number of positive labels in a dataset (prevalence), you still can't settle an F1 ranking just by looking at disagreements. You must know which agreements are actually positive.
  • Shared Completions Matter: By assuming both models are being evaluated against the same underlying ground truth (rather than calculating separate F1 bounds for each), the authors significantly tighten the bounds, making certification much easier.
  • Tolerance is a Superpower: Introducing even a tiny amount of tolerance (e.g., $\epsilon = 0.01$) drastically reduces the number of labels required to make a "safe" choice.
  • Batch-Specific Guarantees: Unlike statistical methods that estimate performance on a general population, this method provides a "hard" guarantee for the specific batch of data currently being audited.

Key Results

The authors demonstrated that their method effectively certifies model choices while minimizing manual labeling.

Fig. 1: Certification after disagreement labeling, unknown prevalence (0 < P < n). Equal-cell means over all 36 cells: 3,200 pairs, or one validation-selected pair per cell. Points are evaluated tolerances; lines guide the eye. Descriptive fractions, not confidence estimates.

As shown in Figure 1, certification rates jump from ~70% (at zero tolerance) to nearly 80% with a small 0.01 tolerance.

The efficiency gains are most visible in the labeling costs: Table 1 shows that using a 'Magnitude' policy with a 0.01 tolerance reduces the total labels needed from 500 (full labeling) to roughly 40-75 labels on average.

Table 1 shows that using a 'Magnitude' policy with a 0.01 tolerance reduces the total labels needed from 500 (full labeling) to roughly 40-75 labels on average.

For validation-selected pairs (Table 1), the "Magnitude" policy with $\epsilon = 0.01$ required only 40.29 labels on average out of a 500-item batch to certify a choice. This represents a massive reduction in human labeling effort compared to a full audit.

Limitations and Future Work

  • Fixed Scope: The current framework assumes binary predictions and "correct" (gold-standard) labels. It does not yet account for label noise or multi-class F1 (macro-F1).
  • Batch Only: The guarantees are strictly for the evaluated batch and do not extend to future data or general population performance.
  • Acquisition Strategy: While the "Magnitude" policy is effective, the authors note it is not necessarily the theoretical lower bound for optimal label acquisition.
  • Computational Assumptions: The method relies on "disagreement-first" labeling, which is a standard but specific workflow constraint.

Summary updated

Abstract

Agreement labels cancel from an accuracy comparison but can change an F1 ranking. We specialize partial identification to an explicit audit after labeling disagreements: two agreement counts give exact bounds, certifying a ranking or a choice within a specified F1 tolerance under every completion. Across 3,200 frozen model pairs, shared completions tighten bounds and a 0.01 tolerance reduces labeling costs. A zero-loss (weak-winner) control attributes most savings to positive tolerance, not ties. Guarantees concern the evaluated batch, not future data.
BibTeX
@misc{fernandezbarrios2026when,
  author = {Roberto Fernández-Barrios and Iker Pastor-López and Asier González-Santocildes and Pablo García Bringas},
  title = {{When Disagreement Labels Do Not Identify F1 Rankings}},
  year = {2026},
  eprint = {2611.01712},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01712},
}
Figure