TL;DR
This paper introduces M-Tuning, a novel prompt tuning method for vision-language models that addresses the "label bias" problem in open-set recognition. By incorporating "open words" from WordNet, M-Tuning simulates an open-set scenario during training, allowing models to distinguish between known and unknown classes more effectively. A Combinatorial Tuning and Testing (CTT) strategy is also proposed to scale this approach to large datasets.
What Problem Does This Paper Address?
Vision-language (VL) models, when fine-tuned using prompt tuning, often exhibit a "label bias." This means that even when presented with images from classes not seen during training (open-set scenarios), these models tend to incorrectly classify them as one of the known training classes. This is a significant hurdle for open-set recognition (OSR), where the goal is to correctly identify an image as belonging to a known class or as an unknown class. Existing OSR methods typically train models from scratch, which is inefficient and doesn't leverage the power of pre-trained VL models. Prompt tuning methods, while parameter-efficient, suffer from this label bias, making them unsuitable for realistic OSR tasks where unknown classes are truly unknown.
Methods
The core of the proposed solution is M-Tuning, which tackles the label bias by:
- Introducing "Open Words": Instead of solely relying on known class names for prompts, M-Tuning incorporates "open words" sourced from WordNet. These words are semantically distant from the training classes. This expands the prompt's vocabulary, allowing it to consider possibilities beyond the known classes and simulate an open-set environment during training.
- Simulated Open-Set Scenario: By including these open words, the model is trained to predict not only from the closed-set classes but also from these "open" concepts. This encourages a separation in the probability distributions between known and unknown classes.
- Combinatorial Tuning and Testing (CTT): For large datasets with many classes, M-Tuning is applied in a "divide-and-conquer" fashion. The dataset is split into smaller groups of classes. M-Tuning is performed independently on each group, and then during testing, predictions from all groups are combined to select the most optimal prompt for classification. This strategy helps manage the complexity of large-scale OSR.
Datasets
The paper evaluates M-Tuning on several standard datasets for OSR tasks:
- Small Datasets: CIFAR10, CIFAR+10, CIFAR+50, CIFAR100, and TinyImageNet.
- Large Datasets: ImageNet-100, ImageNet-200, and ImageNet-LT.
The authors also mention using ImageNet-crop and ImageNet-resize, as well as LSUN-crop and LSUN-resize as unknown data sources for detailed open-set recognition experiments.
Key Insights and Ideas
- Label Bias is a Major Hurdle: The paper strongly emphasizes that the inherent label bias in prompt tuning methods is the primary reason they fail in open-set scenarios.
- Open Words as a "Placeholder" for the Unknown: Introducing semantically distant "open words" acts as a clever proxy for unknown classes, forcing the model to learn to differentiate.
- WordNet is a Valuable Resource: The paper demonstrates that WordNet can be effectively utilized not just for hierarchical information but also to expand the vocabulary for OSR.
- CTT for Scalability: The CTT strategy is crucial for making prompt tuning viable for large, complex datasets, showing that breaking down the problem improves performance.
- Lower Semantic Similarity is Better: A key finding is that open words with lower semantic similarity to the closed-set classes lead to better OSR performance. This suggests that the "unknown" should be as distinct as possible from the "known."
Key Results
M-Tuning consistently outperforms existing prompt-based methods and OSR baselines across various datasets and settings:
- Small Datasets: M-Tuning achieves significantly higher AUROC and closed-set accuracy compared to baselines like CLIP+ZSL, CoOp, and CoCoOp, often by a large margin (e.g., 94.66% AUROC on CIFAR10 in the 16-shot setting).
- Large Datasets: The CTT strategy combined with M-Tuning also shows superior performance on ImageNet-100, ImageNet-200, and ImageNet-LT, with AUROC scores reaching up to 99.01% on ImageNet-100 (16-shot).
- Detailed Open-Set Recognition: In experiments where both known and unknown classes need to be classified, M-Tuning demonstrates robust performance, with known classes confidently predicted and unknown classes receiving much lower probabilities.
- Effectiveness of Open Words: Ablation studies show that M-Tuning without open words performs worse than with them, validating their importance in mitigating label bias.
Limitations and Future Work
- Necessity of WordNet/Database: The method relies on a database like WordNet to source open words. While the paper suggests other databases could be used, the availability and quality of such a database are implicit requirements.
- Optimal Number of Open Words: While M-Tuning is less sensitive to the exact number of open words, finding the optimal number for a given dataset might require some experimentation.
- Future Work: The paper suggests that exploring other databases for open words and further investigating the semantic similarity principle could be fruitful directions. The CTT strategy's effectiveness on even larger and more complex datasets could also be explored.
Summary updated