Abstract
In realistic open-set scenarios where testing data may contain unknown classes, current prompt-tuning methods for vision-language (VL) models often exhibit significant "label bias," misclassifying unknown objects as known training classes with high confidence. We propose M-Tuning (Mitigated Tuning), a regularized prompt-tuning framework designed to address this issue. M-Tuning introduces "open words" from WordNet to simulate an open-set environment during training, allowing the model to reserve probability space for unseen categories. Furthermore, we introduce Combinatorial Tuning and Testing (CTT), a divide-and-conquer strategy that improves performance on large-scale datasets by decomposing complex classification tasks into manageable sub-groups. Our approach achieves state-of-the-art results across various dataset scales, significantly improving unknown detection without sacrificing closed-set accuracy.
The Challenge of Open-Set Recognition
Open-Set Recognition (OSR) is a critical capability for deploying AI in the real world, where models must correctly identify known classes while flagging unknown objects as "unknown." Standard Vision-Language models like CLIP, when fine-tuned using popular methods like CoOp, suffer from extreme label bias. Because these models are optimized only on a fixed set of labels, they attempt to force every input into one of the known categories. For instance, a model trained on "cats" and "dogs" might confidently classify a "tractor" as a dog. This overconfidence makes them unreliable for open-world applications.
Methodology: Mitigating Label Bias
The M-Tuning framework introduces two primary innovations to break this bias:
1. M-Tuning (Mitigated Tuning)
The core idea of M-Tuning is to simulate "openness" during the prompt-tuning phase. Instead of using only the closed-set labels to form prompt texts, the method incorporates "open words" retrieved from WordNet. These words are selected to be semantically distant from the training classes. By including these irrelevant words, the model learns that not every input belongs to the training set, effectively "regularizing" the prompt embeddings.
As shown in the figure above, the text encoder processes both closed-set labels and open words. This shift in the training objective forces the model to produce lower maximum probabilities for open-set data on the closed-set classes, creating a clearer separation between knowns and unknowns.
2. Combinatorial Tuning and Testing (CTT)
OSR performance typically degrades as the number of classes increases, primarily due to a rising false-positive rate. CTT addresses this by splitting large datasets into smaller, independent groups. Each group undergoes its own M-Tuning session. During inference, the model aggregates predictions from all groups and selects the most confident "sub-prompt" to make the final classification or "unknown" call.
Experimental Results
We evaluated M-Tuning against traditional OSR baselines and modern prompt-learning techniques across small-scale (CIFAR) and large-scale (ImageNet) datasets.
Performance Across Scales
M-Tuning consistently outperforms existing methods. On small datasets like CIFAR10, it achieves an AUROC of 96.29% for unknown detection, a massive leap from standard CoOp's 81.22%. On large-scale benchmarks like ImageNet-100, it reaches 98.93% AUROC.
The figure above highlights the effectiveness of the combined approach. Without M-Tuning (a), the model cannot distinguish between open and closed sets (b). Adding M-Tuning (c) creates a bimodal distribution (d) that facilitates OSR. Finally, the CTT strategy (e) significantly boosts closed-set accuracy on large datasets like ImageNet-LT (g) compared to standard tuning (f).
Conclusion
M-Tuning provides a robust and parameter-efficient solution for open-set recognition in vision-language models. By simulating an open-set environment with WordNet-sourced words and employing a divide-and-conquer strategy for large datasets, it effectively mitigates label bias. This allows models to maintain high accuracy on known tasks while gaining the crucial ability to say "I don't know" when encountering the unfamiliar.
Summary updated