Linear Probing of Frozen Foundation Model Representations

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2503 papers539 conceptsfrozen 2026-10-04

When we train a massive foundation model, we are essentially asking it to build a compressed, internal map of a domain—whether that is the English language, the human genome, or the visual world. But how do we know what is actually on that map?

If we "fine-tune" the model by allowing all its weights to change for a specific task, we muddy the waters. We can no longer tell if the model already "knew" how to identify a gene or a sentiment, or if it simply learned to do so during the fine-tuning process. To solve this, researchers use linear probing: they freeze the model’s weights, treat it as a static feature extractor, and train only a tiny, lightweight "head" (usually a single linear layer) on top.

The Intuition of the Read-Only Test

Think of a foundation model as a high-end library. Fine-tuning is like allowing a student to reorganize the entire library to help them pass one specific exam. They might pass, but the library is now biased toward that one subject. Linear probing is like giving the student a "read-only" pass. If they can pass the exam just by looking at the existing organization of the books, it proves the library was already well-structured for that knowledge.

This "frozen" evaluation protocol, as described in AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models, ensures that performance metrics reflect the intrinsic information captured during pre-training. As Gabriele Campanella and colleagues demonstrate in their clinical benchmark of pathology models, this method isolates the representational quality of the encoder from the learning capacity of the downstream architecture.

Probing Across Domains

The power of linear probing is its universality. It allows researchers to compare models trained on completely different data using a consistent "yardstick."

  • Genomics: In the BEND benchmark, Felix Teufel and Frederikke Isa Marin use frozen representation probing to show that while DNA models are getting better at "gene finding," they still struggle with long-range dependencies.
  • Computer Vision: Yunze Man used this technique in Lexicon3D to reveal that unsupervised models like DINOv2 are surprisingly strong all-rounders for 3D scene understanding, even without explicit 3D training.
  • Speech: Maureen de Seyssel found that self-supervised speech models naturally encode a speaker's gender and language, even when never told to do so, by probing their hidden representations.
  • Healthcare: Researchers at Stanford, including Jason Alan Fries, used linear probing to test if a shared foundation model for Electronic Health Records could capture medical knowledge that generalizes across different hospitals.

Why Linear?

Why do we use a simple linear layer instead of a complex neural network as the probe? If the probe is too powerful, it might "learn" the task itself, defeating the purpose of evaluating the frozen features.

A linear probe asks a very specific question: Is this information linearly separable in the latent space? If a simple logistic regression can identify a disease from an embedding, as seen in TraitGym by Gonzalo Benegas, it means the foundation model has done the hard work of untangling the complex raw data into a clear, usable signal.

The Limits of the Prior

Interestingly, linear probing can even reveal the "innate" capabilities of model architectures before they have seen a single data point. Seungdae Baek showed that untrained deep neural networks spontaneously develop face-selective units. By using a linear readout on a randomly initialized network, they proved that some "knowledge" is actually a byproduct of the structural wiring (the "neural language prior") rather than the training data itself, a concept further explored by Dan Busbridge in Neural Language Priors.

By keeping the backbone frozen, linear probing provides a conservative, honest assessment of what a model truly understands, serving as the primary tool for navigating the rapidly expanding landscape of foundation models.

Papers in this version

Showing 60 of 503 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 377 papers
  2. Version 1 · 2026-10-04 · 503 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_linear_probing_ebdb615f,
  title = {linear probing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-probing/rdr_ebdb615f5d944ad79640ef729c07cdee},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_probing_of_frozen_foundation_mode_ebdb615f_v1,
  title = {Linear Probing of Frozen Foundation Model Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-probing-of-frozen-foundation-model-representations/rdr_ebdb615f5d944ad79640ef729c07cdee/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}