When a machine learning model leaves the safety of a curated training set and enters the real world, it encounters a fundamental challenge: it does not know what it does not know. In traditional closed-world evaluation, metrics like standard accuracy, mean Average Precision (mAP), or mean Intersection-over-Union (mIoU) assume a static vocabulary of classes. However, as Yiming Li et al. note in Open World Object Detection: A Survey, these traditional metrics fail to capture how much a model's reliability is compromised when exposed to an unconstrained wilderness of novel data.
The core issue stems from a phenomenon known as metric gaming. Consider an open-set environment where 95% of the test instances belong to known classes and 5% belong to entirely novel, unknown categories. A model that completely ignores the possibility of novelty and blindly classifies every input into a known category can still achieve a stellar 95% accuracy score. Standard arithmetic averaging aggregates performance across classes in a way that masks total failure on unknown categories, creating a misleading perception of robustness (Silvia Bucci et al. in On the Effectiveness of Image Rotation for Open Set Domain Adaptation; Seun-An Choe et al. in Open-Set Domain Adaptation for Semantic Segmentation). To build systems that can safely operate in open environments, the research community had to redesign how performance is measured.
The Mathematical Guardrail of the Harmonic Mean
To prevent models from over-optimizing for known classes at the expense of novelty detection, researchers introduced evaluation protocols anchored by the harmonic mean of seen and unseen class performance (often referred to as the H-score or HOS).
To understand why this works, contrast the arithmetic mean with the harmonic mean. The arithmetic mean allows a high score in one category to compensate for a near-zero score in another:
$$\text{Arithmetic Mean} = \frac{A_{\text{known}} + A_{\text{unknown}}}{2}$$
If a model achieves $100\%$ accuracy on known classes ($A_{\text{known}} = 1.0$) but completely fails to detect any unknown samples ($A_{\text{unknown}} = 0.0$), its arithmetic mean is still a respectable $50\%$.
The harmonic mean, by contrast, is calculated as the reciprocal of the arithmetic mean of the reciprocals:
$$H = \frac{2 \cdot A_{\text{known}} \cdot A_{\text{unknown}}}{A_{\text{known}} + A_{\text{unknown}}}$$
Mathematically, the harmonic mean is strictly bounded by the minimum of the two values. If $A_{\text{unknown}}$ drops to $0$, the entire metric collapses to $0$, regardless of how perfect $A_{\text{known}}$ is.
As Bernt Schiele et al. demonstrated in their foundational sanity check for zero-shot learning, Zero-Shot Learning - The Good, the Bad and the Ugly, this formulation serves as a robust evaluation metric for mixed distributions. It forces the optimization process to prioritize an equilibrium between recognizing established categories and rejecting novel inputs (Mengmeng Jing et al. in Balanced Open Set Domain Adaptation via Centroid Alignment). A system cannot game the evaluation by simply being overly conservative or overly aggressive in its rejections (Fabio Cermelli et al. in Boosting Deep Open World Recognition by Clustering).
Coupling Classification and Detection
While the harmonic mean balances the two objectives, evaluating classification and novelty detection as independent tasks still leaves a subtle loophole. A model might be evaluated on its ability to classify known data in one test, and its ability to flag anomalies in a separate out-of-distribution (OOD) test.
In OpenAUC: Towards AUC-Oriented Open-Set Recognition, Xiaochun Cao et al. exposed that this decoupling allows inconsistent model rankings. A model can appear high-performing on paper even if it fails significantly when classification and detection are required simultaneously.
To close this loophole, the authors introduced a pairwise formulation that couples the two tasks. Under this paradigm, a model is only credited when it correctly identifies a known class and simultaneously ranks an unknown sample as more novel than that specific known sample. This prevents classification-based metrics from being gamed by models that prioritize closed-set accuracy while remaining entirely incapable of rejecting unknown samples.
Hidden Biases and the Mixed-Class Challenge
As open-set evaluation has matured, researchers have uncovered deeper nuances where even standard balanced metrics can distort real-world utility.
One major limitation is the discriminability blindspot. In DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection, Zhi Zhou et al. argued that the standard practice of using the harmonic mean fails to capture a model's ability to differentiate between seen and unseen classes in a realistic, mixed-class environment. If a model evaluates base and new classes in isolation, a high harmonic mean may mask a significant failure in discriminability when the model is presented with a combined, simultaneous set of known and unknown categories.
Furthermore, the metric can become distorted under severe class imbalance. As Zelin Zang et al. observed in Boosting Novel Category Discovery Over Domains with Soft Contrastive Learning and All in One Classifier, standard evaluation metrics for novelty discovery can produce biased results when the distribution of known and unknown samples is highly skewed. If one group significantly outweighs the other, the standard H-score may fail to accurately reflect a model's true ability to generalize across diverse categories without over-emphasizing the majority class.
Ultimately, balanced open-set evaluation is not just about changing a formula; it is about aligning mathematical rewards with the messy, unpredictable reality of deployment. By shifting from simple averages to strict geometric guardrails and coupled pairwise metrics, the field ensures that models are penalized for what they ignore, forcing them to truly master the boundary between the known and the unknown.
Key Foundational Work
Zero-Shot Learning - The Good, the Bad and the Ugly (2017) — contributed 2 concept elements to this direction.
An open-set detection evaluation methodology applied to language and emotion recognition (2007) — contributed 1 concept elements to this direction.