Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have

Elouan Gardès, Seung Eun Yi, Kartik Ahuja, Théo Moutakanni, Huy V. Vo, Piotr Bojanowski, Wolfgang M. Pernice, Loïc Landrieu, Camille CoupriearXiv 2026 · arXiv

TL;DR

FINO (FIne tuning with NO labels) is a novel framework that adapts generic vision foundation models to specialized scientific domains using freely available metadata instead of scarce task labels. By strategically encouraging informative metadata factors and suppressing spurious ones, FINO consistently outperforms fully supervised fine-tuning and matches highly engineered, domain-specific state-of-the-art models across microscopy, medical imaging, and remote sensing.

What Problem Does This Paper Address?

While massive vision foundation models (like DINOv2) excel at natural images, their performance often degrades when applied to specialized scientific fields such as satellite imagery or cellular microscopy. Standard solutions like supervised fine-tuning require large amounts of labeled data, which are notoriously scarce and expensive to obtain in scientific domains. Furthermore, fine-tuning often causes "task-specific collapse," destroying the model's general robustness and making it prone to overfitting on spurious correlations (like batch effects or sensor characteristics).

Crucially, scientific datasets almost always come with rich, structured metadata (e.g., timestamps, geographic coordinates, camera angles, or preparation plates) that are collected automatically and at zero extra cost. This paper addresses a key question: How can we leverage this abundant, free metadata to specialize foundation models without using any task labels?

Methods

The authors introduce FINO, a unified metadata-driven representation adaptation framework. It builds on top of a standard self-supervised learning (SSL) setup (DINO/iBOT) and introduces weak supervision by partitioning metadata into two categories:

  1. Informative factors ($M_+$): Variations the model should preserve because they correlate with meaningful underlying structures (e.g., the specific antibody used in a microscopy stain).
  2. Spurious factors ($M_-$): Nuisance variations or batch effects the model should suppress (e.g., the specific experimental plate or sensor resolution).
The FINO framework leverages metadata as weak supervision alongside self-supervised learning, followed by lightweight probing.

To handle highly granular discrete metadata (spanning thousands of categories) and continuous variables simultaneously, FINO uses two distinct guidance mechanisms:

  • Discrete Metadata Guidance: Instead of standard contrastive learning which struggles to scale, FINO maintains an Exponential Moving Average (EMA) prototype bank. The student network's embeddings are contrasted against these prototypes using an InfoNCE loss.
  • Continuous Metadata Guidance: For continuous values, FINO trains simple MLP predictor heads using a mean squared error (MSE) regression loss.

To eliminate spurious factors, FINO utilizes gradient reversal—effectively forcing the encoder to strip out nuisance information. Finally, an automated gradient-balancing mechanism ensures that heterogeneous metadata losses don't destabilize training.

Detailed look at the FINO training pipeline and discrete metadata prototype bank.

Datasets

FINO was evaluated across four distinct scientific benchmarks:

  • Human Protein Atlas (HPA): Subcellular fluorescence microscopy.
  • Functional Map of the World (FMoW): Satellite imagery for land use classification.
  • iWildCam: Camera trap wildlife monitoring.
  • MIMIC-CXR: Chest X-ray medical imaging.

To test the model's generalizability, the adapted representations were also evaluated on out-of-domain transfer datasets: OpenCell (microscopy), FLAIR-Hub (aerial imagery), and CheXpert (chest X-rays).

Key Insights and Ideas

  • Metadata as an Anchor, Not Just a Nuisance: Traditional domain generalization methods treat metadata purely as a nuisance to be erased. FINO demonstrates that actively encouraging alignment with informative metadata provides a powerful supervisory anchor that structures the representation space without collapsing it.
  • The Danger of Blind Suppression: In scientific data, factors are often entangled. For instance, in microscopy, a "cell line" dictates cell morphology (spurious noise) but also correlates with protein expression (informative signal). The authors show that blindly suppressing such entangled factors hurts performance; positive guidance is often a safer default.
  • Preserving General Feature Space: Because FINO specializes the backbone without task labels, the model avoids task-specific collapse. As visualized via UMAP, the representation space remains cleanly organized by semantic features rather than collapsing into tight, un-generalizable label clusters.
UMAP visualization showing that FINO structures the representation space semantically without the feature collapse seen in supervised baselines.

Key Results

  • Beating Supervised Fine-Tuning: Across all four domains, FINO with a lightweight, frozen probe outperformed fully supervised fine-tuning. For example, on FMoW, FINO achieved a Worst-Group Accuracy (WGA) of 52.9% compared to fine-tuning's 40.5%.
  • Matching Specialized SOTA: FINO matched or exceeded highly engineered, domain-specific state-of-the-art models (such as top Kaggle solutions or massive ensembles) using a single, un-tuned hyperparameter recipe across all datasets.
  • Exceptional Low-Label Performance: In low-supervision regimes (using only 1% to 10% of downstream labels), FINO maintained remarkably high performance, while supervised fine-tuning and Group-DRO performance degraded sharply.
  • Superior Transferability: When transferring a model adapted on HPA to the OpenCell dataset, FINO added a massive +0.122 Adjusted Rand Index (ARI) improvement over the frozen baseline, proving that label-free adaptation preserves cross-dataset utility.

Limitations and Future Work

  • Risk of Reinforcing Biases: If the metadata contains sensitive or unbalanced attributes (e.g., demographic data in medical imaging), treating them as informative factors could inadvertently reinforce systemic biases or lead to disparate performance across subgroups.
  • Dependency on Domain Expertise: While the authors provide an information-theoretic guideline (using asymmetric uncertainty coefficients) to help categorize metadata, partitioning factors into "informative" vs. "spurious" still fundamentally requires domain knowledge.
  • Metadata Quality: The framework assumes metadata is reasonably accurate and available. In settings where metadata is highly incomplete, noisy, or weakly aligned with the target tasks, FINO's effectiveness may decrease.

Summary updated

Abstract

We propose a label-free approach to adapt powerful but generic vision foundation models to specialized scientific domains. Standard supervised fine-tuning is often ill-suited to these settings: labels are scarce, and task-specific training can collapse the model's generality and hurt robustness. We instead leverage metadata to adapt representations to new domains in a self-supervised manner. Our method, FINO, combines a standard self-supervised objective with flexible metadata guidance that handles both highly granular discrete metadata and continuous metadata. It encourages the representation to preserve informative factors while suppressing spurious ones. Across subcellular fluorescence microscopy, Earth observation, wildlife monitoring, and medical imaging, FINO consistently outperforms standard unsupervised domain adaptation and fully supervised adaptation. It also exceeds highly-specialized domain-specific state of the art, while using no task labels for backbone adaptation and only lightweight probes for supervision.
BibTeX
@misc{gardes2026who,
  author = {Elouan Gardès and Seung Eun Yi and Kartik Ahuja and Théo Moutakanni and Huy V. Vo and Piotr Bojanowski and Wolfgang M. Pernice and Loïc Landrieu and Camille Couprie},
  title = {{Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have}},
  year = {2026},
  eprint = {2606.05107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.05107},
}

Loading related content...

Figure