A classifier deployed on a new dataset has two jobs. It must still recognize familiar classes that now look different: a dog drawn as a sketch, or a lesion photographed by another hospital's camera. It must also refuse to name things it has never seen. One-vs-all heads help with the second job. These replace the usual softmax with $K$ independent yes/no classifiers, one per class, so an input can receive "no" from every head and be rejected as unknown. The weak point is the cutoff. A new dataset's visual style lowers every head's confidence at once, so known classes in the new style fall below a cutoff tuned on the training data and get wrongly rejected. The papers that come closest to this setting say so directly. CrossMatch names choosing its rejection threshold without target validation data as an open limitation. MEDIC reports lingering threshold sensitivity under strong style divergence. OVANet avoids the problem only by training on unlabeled target data.
This note makes a narrow claim. One-vs-all heads carry a signal that softmax throws away, and that signal can separate "this looks unfamiliar" from "this is a new class" on a single input, with no target data. Each binary head is trained against a fixed zero boundary, so the heads that say "no" have a meaningful level, which we call the floor. Softmax logits have no such level, because adding a constant to all of them changes nothing. Our hypothesis is that a style shift lowers the whole score profile, floor included. Novelty instead removes the gap between the top head and the floor. We propose a rejection rule built on that gap, choose its one free parameter by holding out whole source datasets, and test whether it closes the gap between deployed and oracle-threshold performance on standard open-set domain generalization benchmarks.
Core Claim
Let $z_1,\dots,z_K$ be the pre-sigmoid scores of the $K$ one-vs-all heads for input $x$. Let $c^*$ be the highest-scoring head. Let $f(x)$ be the median score of the other $K-1$ heads, which is the floor. Our rejection score is
$$ r(x) = \alpha\, z_{c^*} + (1-\alpha)\,\big(z_{c^*} - f(x)\big), $$
and $x$ is rejected as unknown when $r(x) < \tau$.
The two parameters are set on the training data. The mixing weight $\alpha \in [0,1]$ trades absolute confidence against floor-relative margin. The cutoff $\tau$ is set without target data using leave-one-source-dataset-out validation: we train on all source datasets but one and pick $(\alpha, \tau)$ to maximize H-score on the held-out source. The held-out source contains an artificially withheld subset of its classes, which act as unknowns. H-score is the harmonic mean of known-class accuracy and unknown-rejection accuracy.
With $\alpha = 1$ the rule reduces to the usual collective-rejection cutoff. With $\alpha = 0$ it depends only on how far the winning head stands above the floor.
The claim has three parts. First, under style shift, most of the lost H-score in one-vs-all methods comes from a poorly placed threshold, not from poor ranking of known versus unknown inputs. The evidence would be a large gap between the deployed H-score and the H-score at the oracle threshold, alongside a small change in the threshold-free open-set classification rate (OSCR). Second, the floor moves with style while the top-minus-floor margin moves with novelty. Third, the combined rule chosen on a held-out source closes a substantial part of the oracle gap, which raw collective rejection does not.
Relation to Prior Work
Three methods motivate the problem.
- OVANet supplies the architecture we reuse: one-vs-all heads trained on the hardest negative class, reaching an H-score of 53.1 on VisDA. It then adapts to unlabeled target data with entropy minimization. We forbid that access.
- MEDIC trains one-vs-all heads with meta-learning across source splits that differ in both domain and classes. It reports an average PACS H-score of 75.98%, and its authors flag threshold sensitivity as unresolved.
- CrossMatch uses one-vs-all heads to generate pseudo-unknowns from a single source. It admits that its entropy threshold cannot be tuned without target data.
All three change training. None examines what a trained head bank's score profile says about the kind of shift an input has undergone. Our proposal changes only the rejection rule and its selection protocol, so it can be placed on top of any of them.
Two methods constrain the design. DEBUG and SeeCLIP rely on external pretrained models: edge detectors and background removers for DEBUG, and CLIP with Stable Diffusion for SeeCLIP. Their gains are therefore tied to those models. We deliberately use no generator, so any gain we show cannot be credited to synthetic unknowns.
Advancing Open-Set Domain Generalization Using Evidential Bi-Level Hardest Domain Scheduler (EBiL-HaDS) supplies the evaluation protocol: the Office-Home split with 30 held-out classes, OSCR reporting, and a public codebase. Its output is an evidential multi-class model rather than binary heads, so it serves as a non-binary comparison point.
The cross-dataset literature supplies the evaluation stance, not the method. The drop in F1 from 0.788 on ICPR 2012 to 0.482 on ICPR 2014 MITOS-ATYPIA (Mitosis Detection in Breast Cancer Histology Images via Deep Cascaded Networks) shows how badly single-dataset tuning transfers. The same point applies to cutoffs: a cutoff should be judged on a dataset the model never trained on.
The one-vs-all medical work in Open-Set Semi-Supervised Learning for Long-Tailed Medical Datasets was evaluated only within ISIC2018 and ISIC2019 individually. That leaves room for a cross-dataset test.
Research Program
The work has three stages.
Diagnostic stage. We train a plain one-vs-all head bank with hard-negative sampling, following OVANet but without the target stage, and the MEDIC head, on PACS and on the EBiL-HaDS Office-Home split. For every unseen target domain we record three things: the H-score at the source-tuned threshold, the H-score at the oracle target-tuned threshold, and OVANet-style OSCR. We also split the gap between deployed and oracle H-score into known classes wrongly rejected and unknowns wrongly accepted.
Mechanism stage. We test the floor hypothesis directly by comparing the distributions of $f(x)$ and $z_{c^*}-f(x)$ across three groups: known classes in the source style, known classes in the target style, and unknown classes in the target style. The hypothesis predicts that $f(x)$ separates the first two groups, and that the margin separates the last two while barely separating the first two.
Method stage. We apply the rule above with $(\alpha,\tau)$ chosen by leave-one-source-out validation. We report every target domain separately, since average numbers can hide failures on hard styles such as Sketch.
A single-source variant uses style-augmented copies of the source as the "held-out" set. It is reported separately because it is weaker evidence.
The non-vision extension stays within data the file names. We treat ISIC2018 as the source and ISIC2019 as a second dataset. Because ISIC2019 is known to include ISIC2018 images, we remove duplicates before the split. We then withhold several lesion categories as unknowns.
Baselines
The naive reference is a softmax network trained with cross-entropy that rejects inputs whose maximum softmax probability falls below a cutoff. It is the default any practitioner would try. UQGAN argues that its forced-sum outputs make it overconfident on unfamiliar inputs. It is run on the same backbone and training schedule as every other model, and it receives the same leave-one-source-out threshold selection, so the comparison isolates the head type rather than the selection protocol.
The strongest existing method is MEDIC (Generalizable Decision Boundaries). It is the best-reported one-vs-all method for multi-source open-set domain generalization in the file, with an average PACS H-score of 75.98%, compared with 72.95% for DAML and 70.89% for MVDG. We retrain MEDIC with its published recipe and our backbone, and first confirm that we roughly reproduce 75.98%. We then replace only its rejection rule with ours. Beating it means a higher per-domain H-score on at least three of four PACS targets, with no drop in OSCR.
EBiL-HaDS is included as the strongest non-binary competitor, run from its public code. The file gives only relative gains (a 2.45% OSCR improvement on PACS with ResNet18), so the comparison uses our reproduced absolute numbers.
For single-source experiments the reference is DEBUG, which reports a PACS H-score of 54.52%. It is reported as published and not retrained, because it depends on external pretrained modules.
The key ablation keeps the one-vs-all head bank and the leave-one-source-out cutoff but fixes $\alpha = 1$, which is raw collective rejection. If our rule does not beat this ablation, the floor adds nothing and our claim fails.
An oracle cutoff tuned on target labels is reported as a ceiling. We state results as the fraction of the deployed-to-oracle gap that each method closes.
Why not just…?
Why not just tune the cutoff on a held-out source domain? This is the most serious threat, and it is exactly the ablation with $\alpha = 1$. Cross-dataset validation may by itself fix most of the miscalibration that CrossMatch and MEDIC describe. If so, the honest result is that the protocol matters and the score does not. We will report that outcome as the finding, because it would still be the first measurement of how much held-out-source selection recovers for one-vs-all methods.
Why not just use softmax max-logit or a margin between the top two softmax logits? These are already relative scores, so they might capture the same information. The difference is that softmax logits carry no fixed zero, so a style-induced shift in all logits cannot be seen at all. We test this directly by applying the identical top-minus-median rule to softmax logits under the same selection protocol. If it matches the one-vs-all version, the binary heads are not needed for this purpose, and we will say so.
Why not just normalize features using target test-batch statistics? Methods that adapt batch statistics at test time need batches from the target, and they assume the batch is not dominated by unknowns. The adaptation-dependent success of OVANet shows that target access helps. We include test-batch normalization only as a reference that is allowed target access. Our rule works on one input at a time.
Why not just report OSCR and avoid thresholds entirely? A deployed system has to make a yes-or-no decision. OSCR measures ranking, and the diagnostic stage uses it precisely to show that ranking stays roughly intact while the chosen cutoff drifts.
Decisive Checks
We run PACS and DigitsDG with leave-one-domain-out evaluation, the Office-Home split from EBiL-HaDS, and the deduplicated ISIC2018→ISIC2019 pair. Metrics are per-target H-score, known-class accuracy, unknown-rejection accuracy, OSCR, and the fraction of the oracle gap closed. Each result is averaged over three random seeds.
Four outcomes would settle the question.
- Premise check. If the oracle gap for the one-vs-all baselines on PACS is under a few points, threshold miscalibration was never the problem, and the project stops after the diagnostic stage.
- Mechanism check. If style shift does not lower the floor $f(x)$ while novelty leaves it unchanged, the rule has no basis, whatever its benchmark numbers.
- Floor check. The combined rule must close clearly more of the oracle gap than the same one-vs-all heads with $\alpha = 1$ under identical selection. Otherwise the floor adds nothing.
- Head check. The one-vs-all version must beat the softmax top-minus-median version. Otherwise the gain does not require binary heads.
References
- CrossMatch: Cross-Classifier Consistency Regularization for Open-Set Single Domain Generalization
- Generalizable Decision Boundaries: Dualistic Meta-Learning for Open Set Domain Generalization
- OVANet: One-vs-All Network for Universal Domain Adaptation
- Domain Expansion and Boundary Growth for Open-Set Single-Source Domain Generalization
- The Finer the Better: Towards Granular-aware Open-set Domain Generalization
- Advancing Open-Set Domain Generalization Using Evidential Bi-Level Hardest Domain Scheduler
- Mitosis Detection in Breast Cancer Histology Images via Deep Cascaded Networks
- Open-Set Semi-Supervised Learning for Long-Tailed Medical Datasets
- UQGAN: A Unified Model for Uncertainty Quantification of Deep Classifiers trained via Conditional GANs