Towards Realistic Unsupervised Fine-tuning with Vision-Language Models

Jian Liang, Lijun Sheng, Zhengbo Wang, Ran He, Tieniu TanSubmission to ICLR 2024 · Rejected

TL;DR

Existing unsupervised fine-tuning methods for Vision-Language Models (VLMs) like CLIP assume the downstream data strictly matches a predefined list of classes. This paper introduces Unsupervised Universal Fine-Tuning ($U^2$-FT) to handle realistic, messy open-world label shifts and proposes Universal Entropy Optimization (UEO)—a parameter-efficient method that simultaneously boosts in-distribution accuracy and out-of-distribution (OOD) detection without requiring label annotations or sensitivity threshold tuning.

What Problem Does This Paper Address?

Adapting pre-trained vision-language models like CLIP to downstream domains typically relies on labeled data or assumes a clean, closed-set scenario where every unlabeled image belongs to one of the known candidate classes. In real-world deployments, this assumption falls apart: unlabeled data gathered in the wild often contains unseen out-of-distribution classes, and some predefined classes may not appear in the training pool at all.

This paper formalizes this real-world setting as Unsupervised Universal Fine-Tuning ($U^2$-FT), categorizing downstream shifts into four possibilities:

  1. Closed-set: Predefined labels and unlabeled data labels match ($L_u = L_p$).
  2. Partial-set: Unlabeled data only contains a subset of predefined classes ($L_u \subset L_p$).
  3. Open-set: Unlabeled data contains extra unknown classes ($L_p \subset L_u$).
  4. Open-partial-set: Unlabeled data has both missing known classes and unseen novel classes.

Standard fine-tuning (e.g., standard entropy minimization) degrades model reliability in these settings by forcing novel OOD instances into known class prototypes, wrecking the model's ability to reject outliers.

Methods

The authors propose Universal Entropy Optimization (UEO), which balances in-distribution sharpening and out-of-distribution rejection via sample-level confidence:

  • Dual Entropy Objective: UEO weights sample predictions using zero-shot CLIP's maximum softmax probability $w(x)$.
  • For confident samples, it minimizes conditional Shannon entropy, pulling in-distribution representations closer to class text embeddings.
  • For less confident samples (likely OOD), it maximizes marginal entropy over their aggregated, reverse-weighted prediction distribution $\bar{p}$: $$\mathcal{L} = \sum_{x \in \mathcal{B}_t} \tilde{w}(x) H(p(x)) - H(\bar{p}), \quad \text{where} \quad \bar{p} = \sum_{x \in \mathcal{B}_t} \tilde{\Phi}(w(x)) p(x)$$ Here, $\Phi(w)$ is a decreasing function (e.g., $1/w$). Aggregating predictions into $\bar{p}$ before maximizing entropy prevents penalizing difficult in-distribution samples. If weights are uniform, the objective cleanly reduces to standard mutual information maximization.

  • Parameter-Efficient Adaptation: Rather than updating the entire backbone, UEO jointly tunes:

  • Learnable textual prompt vectors (following CoOp) in the language branch.
  • Channel-wise affine transformation parameters within normalization layers (BatchNorm in ResNet-50, LayerNorm in ViT-B/16, inspired by TENT) in the visual branch.
  • Threshold-Free Evaluation: Instead of evaluating open-set classification with threshold-dependent metrics like OS or HOS, the authors evaluate downstream models using per-class ID accuracy (ACC) alongside the Area Under the ROC Curve (AUC) for OOD detection.

Datasets

The benchmark encompasses 15 distinct domains across 4 standard domain adaptation benchmarks:

  • Office (OF): 31 categories across 3 domains (Amazon, DSLR, Webcam).
  • OfficeHome (OH): 65 categories across 4 domains (Art, Clipart, Product, Real-World).
  • VISDA-C (VD): 12 categories across 2 domains (Synthetic to Real).
  • DomainNet (DN): 345 categories across 6 domain styles (Clipart, Infograph, Painting, Quickdraw, Real, Sketch).

For each dataset, the label spaces are systematically divided to simulate closed-set, partial-set, open-set, and open-partial-set shifts during training, while holding an independent mixed ID/OOD test set constant.

Key Insights and Ideas

  • OOD Detection and Adaptation Do Not Have to Conflict: Prior unsupervised methods improved classification accuracy at the direct cost of OOD detection. By weighting marginal entropy by inverse confidence, UEO preserves the feature boundary of unknown classes while refining known classes.
  • Aggregating Before Maximizing Matters: Directly maximizing individual sample entropy for unconfident points risks destabilizing ambiguous in-distribution samples. Maximizing the entropy of the mean prediction batch distribution $\bar{p}$ regularizes predictions toward uniformity safely.
  • Cross-Modality Parameter Tuning: Tuning prompts alone in the text encoder limits transfer capacity. Jointly optimizing affine parameters in the visual normalization layers provides a simple, architecture-agnostic way (compatible with both CNNs and Transformers) to adapt visual features without touching the heavy convolutional or attention weights.

Key Results

  • ResNet-50 Benchmarks:
  • In the complex open-partial shift on Office, UEO achieves 77.2% ACC and 83.2% AUC, making it the only adapted method to surpass zero-shot CLIP (73.2% ACC / 82.8% AUC), while baselines like InfoMax suffered OOD degradation (81.5% AUC).
  • On DomainNet under open-partial shift, UEO achieves 51.9% ACC and 67.4% AUC (compared to CLIP's 47.3% ACC / 66.5% AUC).
  • ViT-B/16 Backbone:
  • Across all four category shift settings on DomainNet, UEO averages 62.0% ACC and 72.7% AUC, outperforming zero-shot CLIP (58.2% / 72.6%), UPL (61.4% / 72.9%), and InfoMax (62.2% / 70.7%, which degrades OOD detection).
  • Conventional Closed-Set Setting:
  • On standard closed-set DomainNet (measuring global accuracy), UEO improves ViT-B/16 zero-shot performance from 57.6% to 61.1%, outperforming UPL (60.6%) and POUF (60.3%).
  • Stability: UEO demonstrates high robustness across varied batch sizes (32 to 128) and loss weighting variations, avoiding brittle hyperparameter tuning.

Limitations and Future Work

  • Task Scope: The study focuses exclusively on image classification. The authors acknowledge that adapting UEO to dense prediction tasks—such as open-vocabulary object detection and semantic segmentation—remains unexplored.
  • Dependence on Zero-Shot Quality: UEO derives its weighting directly from the base model's initial softmax confidence; severely miscalibrated or biased zero-shot predictions in unfamiliar domains could propagate noisy weights during early training steps.

Summary updated

Abstract

The emergence of vision-language models (VLMs), such as CLIP, has spurred a significant research effort towards their application for downstream supervised learning tasks. Although some previous studies have explored the unsupervised fine-tuning of CLIP, they often rely on prior knowledge in the form of class names associated with ground truth labels. In this paper, we delve into a realistic unsupervised fine-tuning scenario by assuming that the unlabeled data might contain out-of-distribution samples from unknown classes. Furthermore, we emphasize the importance of simultaneously enhancing out-of-distribution detection capabilities alongside the recognition of instances associated with predefined class labels. To tackle this problem, we present a simple, efficient, and effective fine-tuning approach called Universal Entropy Optimization (UEO). UEO leverages sample-level confidence to approximately minimize the conditional entropy of confident instances and maximize the marginal entropy of less confident instances. Apart from optimizing the textual prompts, UEO also incorporates optimization of channel-wise affine transformations within the visual branch of CLIP. Through extensive experiments conducted across 15 domains and 4 different types of prior knowledge, we demonstrate that UEO surpasses baseline methods in terms of both generalization and out-of-distribution detection.
BibTeX
@misc{liang2023towards,
  author = {Jian Liang and Lijun Sheng and Zhengbo Wang and Ran He and Tieniu Tan},
  title = {{Towards Realistic Unsupervised Fine-tuning with Vision-Language Models}},
  year = {2023},
  eprint = {2308.12919},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.12919},
}

Loading related content...

Figure