Vision-language models such as CLIP can label images they were never trained on. They do this by comparing each image's embedding to the embedding of a text prompt like "a photo of a dog." These zero-shot labels are noisy for a known geometric reason. Image and text embeddings sit in two separate regions of the shared space, a separation called the modality gap, so a text prompt is only an off-center stand-in for the cluster of images it names. A popular fix is to treat the zero-shot labels as pseudo-labels (machine-generated guesses used as training targets) and train a classifier directly among the image embeddings. That classifier, however, is trained on the same labels it is supposed to correct. It can memorize the confidently wrong ones and repeat them back. This is the confirmation-bias loop that the pseudo-labeling literature warns about.
We propose to let a classical statistical classifier judge each pseudo-label only on examples it did not train on. Frozen CLIP image features and text-derived pseudo-labels feed a bagged ensemble of linear max-margin classifiers. Bagging means each member is trained on a random bootstrap resample of the data. Every image is then re-predicted only by the members whose resample left it out, which gives its "out-of-bag" vote. When that vote disagrees with the text-derived label, the label is revised. The same out-of-bag votes also re-estimate the class frequencies, so the method does not have to assume balanced classes. The decoupled classifier is useful here because of how it is trained, not because it is more flexible. Its held-out judgments come free from bagging and cannot memorize the error under test. The whole procedure runs on cached features on a CPU.
Core Claim
The closest prior work is Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP (InMaP), which supplies both the setting and the bound we target. InMaP builds zero-shot pseudo-labels from text, cleans them with Sinkhorn optimal transport (a procedure that forces predicted class counts to match an assumed class-frequency prior, uniform in their case) and a confidence threshold. It then fits one learnable class centroid per class inside the image space by gradient descent. The paper proves that the error of these image-space centroids is bounded by the pseudo-label error rate.
Our claim is that the remaining pseudo-label errors after InMaP-style cleaning are concentrated among confidently wrong images. An in-sample gradient-trained head cannot detect these errors, because it has fit them. Held-out disagreement between an image-space classifier and the cross-modal label can detect them. Driving down the quantity in InMaP's bound this way should raise accuracy, most of all on long-tailed data where the uniform Sinkhorn prior is wrong.
This differs from InMaP in two concrete ways. First, InMaP never evaluates a pseudo-label with a model that did not train on it; we always do. Second, InMaP fixes the class prior to uniform, whereas we estimate it from out-of-bag votes.
This is not a generic "swap softmax for an SVM or XGBoost" proposal. We choose linear members deliberately. Cross-Modal Mapping: Mitigating the Modality Gap for Few-Shot Image Classification found that adding non-linear projections on CLIP features hurts alignment while a residual linear map helps. That constrains the design: tree ensembles, the usual choice in hybrid pipelines, are included only as a comparison, and we expect them to lose.
Research Program
Setup. The setting is transductive zero-shot classification, as in InMaP. We have a frozen CLIP encoder, class names, and the unlabeled target images. Images are encoded once. Initial soft labels come from cosine similarity against text prompts and are passed through Sinkhorn with a uniform prior, so round zero reproduces InMaP's label cleaning.
Rounds. In each round we fit an ensemble of $B$ linear support vector machines. A support vector machine (SVM) finds the boundary with the widest margin between classes. Each member is trained on a bootstrap resample of the currently accepted pseudo-labeled images, using a squared-hinge loss and a regularization strength chosen by out-of-bag agreement rather than any true label. For image $i$ with image embedding $z_i$, the out-of-bag vote for class $c$ is
$$v_{ic} = \frac{1}{|O_i|}\sum_{b \in O_i} \mathbb{1}[h_b(z_i) = c],$$
where $O_i$ is the set of members whose resample excluded $i$ and $h_b$ is member $b$'s prediction. Image $i$ keeps its label $\hat y_i$ if $v_{i\hat y_i}$ is the largest vote. It is relabeled if another class wins by a margin $\gamma$, and it is set aside as ambiguous otherwise.
Prior re-estimation. The class counts implied by the out-of-bag votes become the marginal for the next round's Sinkhorn step, replacing the uniform prior. A few rounds, typically three to five, give the final labels. The final prediction is the full ensemble's vote.
Built-in diversity. Bagging supplies the classical tri-training ingredient that Tri-net for Semi-Supervised Deep Learning had to recreate artificially. Tri-net needed noise injected into targets and dropout stability checks because neural heads on a shared backbone collapse into identical solutions. Bootstrap resamples differ by construction.
Cost. The cost argument is concrete. Uncertainty estimates based on Monte Carlo dropout (running the network several times with random units switched off), as in FPL+, or on many augmented views per test image, as in Test-Time Prompt Tuning, require repeated forward passes. Out-of-bag votes require one encoder pass per image.
Analysis. Alongside accuracy we study where the corrected errors lie. Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification showed that detection pseudo-label noise piles up near a decision threshold. We ask the analogous question for zero-shot labels: do the images that the out-of-bag vote flags cluster where text proxies sit farthest from their image clusters? We measure this per class as the distance between the text embedding and the centroid of correctly labeled images, using ground-truth labels for analysis only.
As a secondary, cheap probe of the gap geometry, we add each image's deviation from the mean image-minus-text offset studied in Mind the Gap as a candidate uncertainty score. We compare it against out-of-bag disagreement. We expect it to be weaker, because a single global offset vector does not encode which class a text proxy mislabels.
Baselines
Naive reference. The do-nothing reference is vanilla CLIP zero-shot on the same backbone. InMaP reports 77.02% ImageNet top-1 for vanilla CLIP with ViT-L/14@336.
Strongest competitor. The strongest existing method is InMaP itself. It is the most direct competitor because it uses the same inputs (frozen features plus unlabeled target images), the same cleaning at round zero, and the same transductive protocol. It reports 80.21% ImageNet top-1 with ViT-L/14@336 in under a minute on one V100, and more than 5% average improvement over the baseline across 13 downstream tasks with ViT-B/16. Beating it means exceeding its accuracy on the same backbone and the same unlabeled pool in each benchmark, with gains concentrated in the long-tailed variants. Wherever we cannot reproduce 80.21%, we compare against our own rerun under identical feature caches.
Test-time adaptation reference. Test-Time Prompt Tuning is the reference that adapts text rather than image space. It reports a 6.9% gain over zero-shot CLIP on ImageNet-A, a 3.6% average gain across distribution-shift sets, and 0.25s versus 0.10s latency per image. We compare to it on ImageNet-A, -R, -V2 and -Sketch. We note that it is per-sample while we are transductive, so we report both accuracy and total compute.
Classical label-propagation reference. A k-nearest-neighbor relabeler in image space that excludes each image from its own neighborhood is the classical competitor. It is also held-out by construction.
Few-shot reference. Cross-Modal Mapping uses true labels and reports a 1.06% average top-1 gain over 11 datasets. It is not a fair competitor, but it is a useful ceiling-style reference. We run its linear residual map trained on our pseudo-labels instead of true labels.
Fairness conditions. All methods share cached features, prompts, and backbone. Every hyperparameter is tuned only with label-free criteria (out-of-bag agreement, or the method's own internal objective) under an equal wall-clock budget.
Ablations. The decisive ablation keeps the exact bagged SVM ensemble but relabels using in-sample votes instead of out-of-bag votes. A second ablation replaces SVMs with k-fold cross-validated logistic regression, which isolates held-out judgment from the max-margin loss. A third keeps the uniform prior throughout, which isolates prior re-estimation. A fourth swaps linear members for random forests and gradient-boosted trees.
Why not just…?
Why not just run InMaP with a stricter confidence threshold? Tightening the threshold removes uncertain labels. The errors we target are the confident ones, which a threshold keeps by definition. We test this directly by measuring, on the subset of images whose zero-shot confidence is high but whose label is wrong, the detection rate of threshold sweeps versus out-of-bag disagreement. If thresholding recovers most of our gain, the claim fails.
Why not just use leave-self-out k-nearest-neighbor label propagation? This is a real threat. It shares the held-out property and is equally cheap. Neighbor votes in CLIP space may already capture most of the image-space structure our ensemble exploits. Our bet is that neighborhoods inherit the cone-shaped crowding of CLIP embeddings described in Mind the Gap, while margin classifiers use global class structure. If k-nearest-neighbor relabeling matches us within noise across the 13 tasks, we will report that the active ingredient is held-out relabeling in image space, not the classifier family, and the note's contribution narrows accordingly.
Why not just fine-tune the prompts or encoder instead? Prompt tuning moves text toward images and pays backpropagation through the text encoder per sample. Intra-Modal Proxy Learning argues that the optimal classifier lives in image space, so text proxies remain approximations. Fine-tuning the encoder on noisy pseudo-labels is exactly the high-capacity, in-sample fitting that confirmation bias exploits. The comparison against prompt tuning on the shift benchmarks settles this empirically.
Why not just use gradient-boosted trees, as hybrid pipelines usually do? Hybrid successes such as Deep Learning for Hate Speech Detection in Tweets (F1 from 0.753 to 0.930 with embeddings plus gradient-boosted trees) and Object Recognition from very few Training Examples (99.3% patch accuracy with a random forest) used task-trained or convolutional features. Contrastive CLIP features appear to favor linear boundaries, per Cross-Modal Mapping. The tree ablation reports the outcome either way.
Proposed Experiments
We evaluate on ImageNet, CIFAR-100, and the 13 transfer tasks used by InMaP, including Stanford Cars, DTD, and SUN397, with ViT-B/16 and ViT-L/14@336. We add the distribution-shift sets ImageNet-A, -R, -V2 and -Sketch.
To test the prior re-estimation, we build long-tailed versions of each unlabeled pool by subsampling classes geometrically at imbalance ratios of 10 and 100. This is where uniform Sinkhorn is expected to fail. We have not found a result on Lacuna for InMaP on long-tailed pools, so we rerun it there ourselves.
Metrics. We report top-1 accuracy and balanced accuracy, plus the pseudo-label error rate after each round, since this is the quantity in InMaP's bound. We report the area under the ROC curve of each noise score (out-of-bag disagreement, entropy, confidence, gap-offset deviation) for detecting wrong pseudo-labels, overall and on the confident-wrong subset. Finally we report wall-clock time on CPU versus a V100.
Decisive checks. The idea is confirmed only if three things hold. First, out-of-bag relabeling lowers the pseudo-label error rate relative to the identical ensemble using in-sample votes, and this translates into higher accuracy than InMaP on the balanced benchmarks. Second, the gain widens on the long-tailed pools and disappears when prior re-estimation is ablated. Third, out-of-bag disagreement detects confident-wrong labels with clearly higher area under the curve than entropy or confidence.
If the in-sample and out-of-bag variants tie, held-out judgment is not doing the work and the proposal is wrong. If leave-self-out k-nearest-neighbor matches the ensemble, we report the narrower finding described above.
References
- Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP
- Cross-Modal Mapping: Mitigating the Modality Gap for Few-Shot Image Classification
- Tri-net for Semi-Supervised Deep Learning
- FPL+: Filtered Pseudo Label-Based Unsupervised Cross-Modality Adaptation for 3D Medical Image Segmentation
- Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models
- Combating Noise: Semi-supervised Learning by Region Uncertainty Quantification
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
- Deep Learning for Hate Speech Detection in Tweets
- Object Recognition from very few Training Examples for Enhancing Bicycle Maps