Abstract: The emergence of vision-language models (VLMs), such as CLIP, has spurred a significant research effort towards their application for downstream supervised learning tasks. Although some previous studies have explored the unsupervised fine-tuning of CLIP, they often rely on prior knowledge in the form of class names associated with ground truth labels. In this paper, we delve into a realistic unsupervised fine-tuning scenario by assuming that the unlabeled data might contain out-of-distribution samples from unknown classes. Furthermore, we emphasize the importance of simultaneously enhancing out-of-distribution detection capabilities alongside the recognition of instances associated with predefined class labels. To tackle this problem, we present a simple, efficient, and effective fine-tuning approach called Universal Entropy Optimization (UEO). UEO leverages sample-level confidence to approximately minimize the conditional entropy of confident instances and maximize the marginal entropy of less confident instances. Apart from optimizing the textual prompts, UEO also incorporates optimization of channel-wise affine transformations within the visual branch of CLIP. Through extensive experiments conducted across 15 domains and 4 different types of prior knowledge, we demonstrate that UEO surpasses baseline methods in terms of both generalization and out-of-distribution detection.
Introduction
Vision-Language Models (VLMs) like CLIP have revolutionized computer vision with their zero-shot capabilities. However, achieving peak performance on specific downstream tasks often requires fine-tuning. While supervised fine-tuning is effective, it requires expensive labeled data. Unsupervised Fine-Tuning (UFT) offers a more scalable alternative but faces a significant hurdle: "in-the-wild" data is rarely clean.
Most existing UFT methods assume that the unlabeled training set perfectly aligns with a predefined list of classes. In reality, such data often contains Out-of-Distribution (OOD) samples—images from classes the model isn't intended to recognize. Standard fine-tuning techniques like entropy minimization can backfire in these scenarios, forcing the model to assign known labels to unknown objects and destroying its ability to detect OOD samples.
Unsupervised Universal Fine-Tuning ($U^2$-FT)
To address these real-world complexities, this research introduces the Unsupervised Universal Fine-Tuning ($U^2$-FT) framework. This setup requires models to simultaneously improve their accuracy on known classes (generalization) and their ability to identify unknown classes (OOD detection).
Universal Entropy Optimization (UEO)
The proposed solution, Universal Entropy Optimization (UEO), manages the uncertainty of unlabeled data by using CLIP’s own zero-shot confidence as a guide. The method is built on two pillars:
- Dual Entropy Optimization: UEO applies a sample-level weighting strategy. For images where CLIP is already confident (likely known classes), the model minimizes entropy to sharpen its predictions. For uncertain images (likely OOD), it maximizes entropy, pushing the prediction toward a uniform distribution. This prevents the model from "hallucinating" known labels for unknown objects.
- Parameter-Efficient Tuning: Instead of updating the entire model, UEO tunes a small set of parameters: learned textual prompts for class names and visual affine transformations (scaling and shifting) within the image encoder's normalization layers. This dual-modality adaptation bridges the vision-language gap more effectively than tuning either branch alone.
Experimental Evaluation
The authors validated UEO across 15 domains using benchmarks like OfficeHome, DomainNet, and ImageNet. They tested four category shift scenarios, ranging from "Closed-set" (all data known) to the highly realistic "Open-partial" shift (data contains both missing known classes and extra unknown classes).
Key Results
- Superior Balance: Performance was measured using the HAA metric (the harmonic mean of Accuracy and AUC). UEO consistently outperformed baselines like UPL and POUF. For example, on DomainNet under the Open-Partial shift, UEO reached an HAA of 57.1%, compared to CLIP's 53.0%.
- Robustness: UEO is less sensitive to hyperparameters like batch size or specific loss weights, making it more practical for real-world deployment where labeled validation data is unavailable.
- Synergy: The results confirm that simultaneously optimizing for both known and unknown classes leads to a more robust model. As shown in the experimental plots, there is a clear positive correlation between classification success and OOD detection capability.
Conclusion and Future Work
UEO demonstrates that CLIP can be effectively fine-tuned on messy, unlabeled data by intelligently leveraging its internal confidence scores. By balancing entropy minimization and maximization, UEO improves task-specific performance without sacrificing the model's ability to say "I don't know." Future research will look into applying this universal optimization strategy to dense prediction tasks like object detection and semantic segmentation, as well as exploring its performance on non-image modalities.
Summary updated