AutoThreshold: Joint Confusion-Profile Learning for Adaptive Mask Binarization

Shaokai Wu, Xunchu Zhou, Pinzhi Wang, Bo Dong, Xiao Cui, Qi Kong, Yue Ding, Hongtao LuarXiv 2026 · arXiv

TL;DR

AutoThreshold is a post-hoc method that makes image segmentation more robust by predicting an image's entire "confusion profile"—the expected true and false positives across all possible thresholds. This allows a frozen model to adaptively pick the best threshold for any metric (like Dice or $F_2$) on the fly, significantly outperforming fixed thresholds and standard heuristics like Otsu's method when data distributions shift.

What Problem Does This Paper Address?

Most computer vision models for segmentation produce a "score map"—a grid of continuous values representing confidence. However, real-world systems need a binary mask. The standard approach is to pick a fixed threshold (usually 0.5) during validation and stick with it.

The problem? A fixed threshold is brittle. If the lighting changes, the camera degrades, or the model encounters a new domain, the distribution of these scores shifts. A threshold that worked perfectly on the training set might suddenly produce masks full of "noise" or miss the target entirely.

While adaptive methods like Otsu’s thresholding exist, they are often simple heuristics that don't "learn" from data. On the other hand, existing machine learning approaches often try to regress a single "optimal" threshold value, which ignores the broader "utility landscape"—the fact that being slightly off in one direction might be much worse than the other depending on your target metric.

Figure 1: Design space for threshold policies. Scored outputs admit different threshold types, spatial supports (image, patch, or dense location), and decision-unit utilities. We instantiate image-level mask binarization; other supports are valid only when both operator and utility decompose accordingly. Color denotes confidence, and size illustrates a second thresholdable attribute.

Methods

The authors introduce AutoThreshold, a lightweight, "detached" policy head that sits on top of a frozen segmentation model. Instead of predicting a single number, it predicts a Joint Confusion Profile.

Key Innovations:

  1. Joint Target: The model predicts the joint distribution of positive and negative pixels across different score bins. If you have 128 bins, it predicts 256 values representing the "mass" of true foreground and background pixels in each bin.
  2. Cumulative Decoding: By taking the cumulative sum of these predicted masses, the system reconstructs monotone curves for True Positives (TP) and False Positives (FP) across all possible thresholds. This "monotone" property is a crucial inductive bias—as you raise the threshold, you can't suddenly have more positive pixels.
  3. Metric Retargeting: Because the model predicts the underlying confusion statistics (TP, FP, FN), you can calculate any $F_\beta$ metric (like Dice, $F_{0.5}$, or $F_2$) from the same prediction. You just "plug in" a different decoder at the end to find the threshold that maximizes your specific goal.
  4. Label-Free Features: At inference time, the policy head only needs statistics from the score map itself (like histograms, quantiles, and entropy). It doesn't need the ground truth labels it was trained on.

Datasets

The authors evaluated AutoThreshold across two challenging domains:

  • Medical Imaging (Polyp Segmentation): They trained on Kvasir-SEG and tested transfer performance on four external datasets: CVC-300, CVC-ClinicDB, CVC-ColonDB, and ETIS-LaribPolypDB. This tests how well the method handles domain shift between different hospitals and equipment.
  • Autonomous Driving (Road Detection): Using the KITTI Road dataset, they performed a "leave-one-corruption-family-out" (LOCO) evaluation. They trained on various synthetic corruptions (like blur or brightness changes) and tested on entirely unseen types of image degradation.

Key Insights and Ideas

  • Structure Over Scalars: Predicting the entire confusion profile provides a much richer supervision signal than just regressing a single threshold value. It forces the model to understand why a threshold is good, not just what it is.
  • Detachment is a Feature: Because the policy head is detached, you can add it to any existing, frozen segmentation pipeline without retraining the heavy base model.
  • The "Utility Landscape": The method excels when the optimal threshold is "narrow"—meaning a small error in threshold selection leads to a large drop in performance. By predicting the whole curve, AutoThreshold is better at finding these sharp peaks.
  • Auditable Invariants: The use of softmax normalization and cumulative sums ensures the predicted confusion counts are physically plausible (non-negative and monotone), which simple regression doesn't guarantee.

Key Results

AutoThreshold (referred to as "Joint" in the paper) consistently outperformed both fixed thresholds and the popular Otsu's method:

  • Polyp Transfer: On the reference pipeline, "Joint" achieved a macro Dice of 0.6324, which is 0.0085 higher than Otsu and significantly better than a fixed global threshold (0.6080).
  • Robustness to Corruption: In the KITTI Road experiments, it improved over standard scalar regression by 0.0082 Dice on unseen corruptions.
  • Metric Flexibility: The same model checkpoint successfully optimized for $F_{0.5}$, Dice, and $F_2$ simply by changing the final argmax calculation, outperforming Otsu across all three.
Figure 2: True and predicted Dice profiles for representative external images. Vertical lines mark thresholds from source Global, scalar regression, Joint, and the sampled Oracle. The profile supplies a maximizer rather than a calibrated utility estimate; the right case exposes a remaining selection error.

Limitations and Future Work

  • Decomposability: The current formulation requires the evaluation metric (like Dice) to be "decomposable" at the same level as the threshold policy. It works for image-level Dice, but wouldn't directly apply to complex corpus-level metrics like COCO Average Precision (AP) without modification.
  • Image-Level Focus: The paper focuses on picking one threshold per image. While the math supports patch-level or pixel-level thresholds, the authors note that this requires the underlying operator (the segmenter) to be regionalized as well.
  • Bayes Optimality: The authors clarify that while their "plug-in" rule is highly effective, they don't claim it is theoretically Bayes-optimal for non-linear metrics like $F_\beta$. Exploring the gap between this structured approach and direct value learning remains an open area for research.

Summary updated

Abstract

A single validation-tuned threshold can become brittle when dense-score distributions shift. We introduce AutoThreshold, a post-hoc policy that predicts the joint mass of positive and negative pixels over score bins. This target directly parameterizes threshold-indexed confusion counts, avoiding products of independently estimated prevalence and class-conditional rates. Softmax normalization and cumulative decoding yield coherent, monotone confusion profiles; a plug-in decoder then selects a threshold for any $F_β$ objective without retraining the head. Across three independently trained Kvasir pipelines transferred to four polyp datasets, Joint exceeds continuous Otsu by $0.0080 \pm 0.0046$ external macro Dice, with positive gains in 3/3 runs. On the reference pipeline, three head seeds achieve $0.6324 \pm 0.0027$, 0.0085 above Otsu; direct-utility Ridge achieves 0.6342, with the paired difference interval spanning zero. The same Joint checkpoints outperform Otsu for $F_{0.5}$, Dice, and $F_2$. On developmental KITTI Road leave-one-corruption-family-out evaluation, Joint improves over scalar regression by 0.0082 [0.0064, 0.0101], while direct utility is 0.0022 higher. These results establish joint confusion prediction as a useful, metric-retargetable inductive bias, without claiming Bayes-optimal $F_β$ decisions.
BibTeX
@misc{wu2026autothreshold,
  author = {Shaokai Wu and Xunchu Zhou and Pinzhi Wang and Bo Dong and Xiao Cui and Qi Kong and Yue Ding and Hongtao Lu},
  title = {{AutoThreshold: Joint Confusion-Profile Learning for Adaptive Mask Binarization}},
  year = {2026},
  eprint = {2611.14279},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14279},
}
Figure