R-Tuning: Regularized Prompt Tuning in Open-Set Scenarios

Ning Liao, Xiaopeng Zhang, Min Cao, Qi Tian, Junchi Yan

Abstract

In realistic open-set scenarios where testing data may contain unknown classes, current prompt-tuning methods for vision-language (VL) models often exhibit significant "label bias," misclassifying unknown objects as known training classes with high confidence. We propose M-Tuning (Mitigated Tuning), a regularized prompt-tuning framework designed to address this issue. M-Tuning introduces "open words" from WordNet to simulate an open-set environment during training, allowing the model to reserve probability space for unseen categories. Furthermore, we introduce Combinatorial Tuning and Testing (CTT), a divide-and-conquer strategy that improves performance on large-scale datasets by decomposing complex classification tasks into manageable sub-groups. Our approach achieves state-of-the-art results across various dataset scales, significantly improving unknown detection without sacrificing closed-set accuracy.

The Challenge of Open-Set Recognition

Open-Set Recognition (OSR) is a critical capability for deploying AI in the real world, where models must correctly identify known classes while flagging unknown objects as "unknown." Standard Vision-Language models like CLIP, when fine-tuned using popular methods like CoOp, suffer from extreme label bias. Because these models are optimized only on a fixed set of labels, they attempt to force every input into one of the known categories. For instance, a model trained on "cats" and "dogs" might confidently classify a "tractor" as a dog. This overconfidence makes them unreliable for open-world applications.

Methodology: Mitigating Label Bias

The M-Tuning framework introduces two primary innovations to break this bias:

1. M-Tuning (Mitigated Tuning)

The core idea of M-Tuning is to simulate "openness" during the prompt-tuning phase. Instead of using only the closed-set labels to form prompt texts, the method incorporates "open words" retrieved from WordNet. These words are selected to be semantically distant from the training classes. By including these irrelevant words, the model learns that not every input belongs to the training set, effectively "regularizing" the prompt embeddings.

M-Tuning Pipeline: Incorporating open words from WordNet to regularize the vision-language joint learning process.

As shown in the figure above, the text encoder processes both closed-set labels and open words. This shift in the training objective forces the model to produce lower maximum probabilities for open-set data on the closed-set classes, creating a clearer separation between knowns and unknowns.

2. Combinatorial Tuning and Testing (CTT)

OSR performance typically degrades as the number of classes increases, primarily due to a rising false-positive rate. CTT addresses this by splitting large datasets into smaller, independent groups. Each group undergoes its own M-Tuning session. During inference, the model aggregates predictions from all groups and selects the most confident "sub-prompt" to make the final classification or "unknown" call.

Experimental Results

We evaluated M-Tuning against traditional OSR baselines and modern prompt-learning techniques across small-scale (CIFAR) and large-scale (ImageNet) datasets.

Performance Across Scales

M-Tuning consistently outperforms existing methods. On small datasets like CIFAR10, it achieves an AUROC of 96.29% for unknown detection, a massive leap from standard CoOp's 81.22%. On large-scale benchmarks like ImageNet-100, it reaches 98.93% AUROC.

Comparison of tuning strategies: (a-d) shows how M-Tuning shifts probability distributions to better separate open and closed sets. (e-g) demonstrates the impact of CTT on large-scale dataset accuracy.

The figure above highlights the effectiveness of the combined approach. Without M-Tuning (a), the model cannot distinguish between open and closed sets (b). Adding M-Tuning (c) creates a bimodal distribution (d) that facilitates OSR. Finally, the CTT strategy (e) significantly boosts closed-set accuracy on large datasets like ImageNet-LT (g) compared to standard tuning (f).

Conclusion

M-Tuning provides a robust and parameter-efficient solution for open-set recognition in vision-language models. By simulating an open-set environment with WordNet-sourced words and employing a divide-and-conquer strategy for large datasets, it effectively mitigates label bias. This allows models to maintain high accuracy on known tasks while gaining the crucial ability to say "I don't know" when encountering the unfamiliar.

Summary updated

Abstract

In realistic open-set scenarios where labels of a part of testing data are totally unknown, current prompt methods on vision-language (VL) models always predict the unknown classes as the downstream training classes. The exhibited label bias causes difficulty in the open set recognition (OSR), by which an image should be correctly predicted as one of the known classes or the unknown one. To learn prompts in open-set scenarios, we propose the Regularized prompt Tuning (R-Tuning) to mitigate the label bias. It introduces open words from the WordNet to extend the range of words forming the prompt texts from only closed-set label words to more. Thus, prompts are tuned in a simulated open-set scenario. Besides, inspired by the observation that classifying directly on large datasets causes a much higher false positive rate than on small datasets, we propose the Combinatorial Tuning and Testing (CTT) strategy for improving performance. CTT decomposes R-Tuning on large datasets as multiple independent group-wise tuning on fewer classes, then makes comprehensive predictions by selecting the optimal sub-prompt. For fair comparisons, we construct new baselines for OSR based on VL models, especially for prompt methods. Our method achieves the best results on datasets with various scales. Extensive ablation studies validate the effectiveness of our method.
BibTeX
@article{liao2025r,
  author = {Ning Liao and Xiaopeng Zhang and Min Cao and Qi Tian and Junchi Yan},
  title = {{R-Tuning: Regularized Prompt Tuning in Open-Set Scenarios}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2025},
  doi = {10.1109/tcsvt.2024.3524181},
}
Figure