What Linear Probes Miss: Multi-View Probing for Weight-Space Learning

Eunwoo Heo, Kyeongkook Seo, Jaejun Yoo

The "Model Jungle" and the Limits of Linear Probing

The explosive growth of open-source model repositories has created a "Model Jungle"—a landscape where millions of checkpoints are shared, often without adequate documentation or metadata. Determining the training data or generalization capabilities of a fine-tuned ResNet or a Stable Diffusion LoRA adapter typically requires expensive evaluations.

Weight-space learning offers a compelling alternative by treating a model's weights directly as data to predict its properties. However, because neural network weights are massive and high-dimensional, processing them in full is computationally prohibitive. To make this tractable, researchers rely on "probing." This technique injects small, learnable vectors into a weight matrix to extract permutation-equivariant representations based on the matrix's response.

Unfortunately, existing probing methods are limited by a single-view design, typically only capturing how rows of a weight matrix interact with a probe. The authors prove that this single-view approach creates a fundamental ambiguity problem: two completely different weight matrices can produce the exact same probe response, making them indistinguishable. Furthermore, these linear probes capture only first-order structures, missing the rich, higher-order correlation patterns inherent in row–column interactions.

Introducing MVProbe: A Multi-Perspective Framework

To bridge this gap, the paper introduces MVProbe, a multi-perspective probing framework that synthesizes first-order signals with interaction-aware, second-order (Gram-based) views. MVProbe analyzes a weight matrix $X$ through four complementary branches:

  1. Row Probing ($XU$): The standard first-order approach.
  2. Column Probing ($X^T V$): Captures the input-side geometry.
  3. Row Kernel Probing ($XX^T W$): A second-order view analyzing relationships between output neurons.
  4. Column Kernel Probing ($X^T X Z$): A second-order view analyzing correlations between input features.

By adding these second-order branches, MVProbe overcomes the ambiguity problem. The authors provide a formal proof (Theorem 4.1) demonstrating that higher-order probes are a theoretical necessity for distinguishability. Geometrically, probe vectors act as "landmark" directions in a high-dimensional space; while first-order probes define directions in the ambient space, second-order probes define landmark combinations in a kernel space.

Overcoming Computational and Scaling Hurdles

Implementing second-order probing introduces two major challenges: computational cost and scale imbalance.

  • Efficiency via Associativity: Computing a full Gram matrix ($XX^T$) is notoriously slow. MVProbe bypasses this using the associative property of matrix multiplication—computing $X(X^T W)$ instead of $(XX^T)W$. This keeps the computational complexity identical to simple linear probing.
  • Per-Sample Standardization: Mathematically, second-order signals are significantly larger in magnitude than first-order ones. Without intervention, they would drown out the linear signals. The authors derived a principled standardization strategy based on the scaling laws of different probing orders, ensuring balanced contributions from all four branches.

Key Findings and State-of-the-Art Results

MVProbe was evaluated on the Model Jungle benchmark across diverse architectures, including ResNet-101, SupViT, MAE, and DINO, as well as on Stable Diffusion LoRA adapters.

  • State-of-the-Art Performance: MVProbe consistently outperformed the previous leading method, ProbeX, achieving +5.08% accuracy on ResNet and +5.04% on DINO.
  • Preventing Collapse at Scale: On the challenging 1,000-class Stable Diffusion task (SD1k), ProbeX suffered a catastrophic collapse, dropping to 35.75% accuracy. In contrast, MVProbe maintained a highly robust 97.88% accuracy.
  • Efficiency Maintained: Despite processing four times the branches and utilizing higher-order math, MVProbe’s training time per epoch remains nearly identical to simpler methods (e.g., 1.375s vs. 1.278s).
  • Architectural Insights: The study revealed that in Transformers, the "Value" ($W_V$) matrices are the most informative for weight-space learning, likely serving as the primary repository for task-relevant semantic features. Furthermore, MVProbe proved remarkably stable across different network depths, unlike previous methods that rely on selecting specific "lucky" layers.

Limitations and Future Directions

While MVProbe represents a significant leap forward, absolute accuracy on self-supervised models (like MAE and DINO) remains lower than on supervised models. Additionally, the current framework focuses primarily on single-layer probing to maximize efficiency. Future research could explore multi-layer aggregation to combine signals across the entire network, or develop architecture-aware branch selection to further enhance performance on self-supervised backbones.

Summary updated

Abstract

The explosive growth of open-source model repositories has created a Model Jungle, where checkpoints are frequently shared without adequate documentation or metadata. While weight-space learning offers a pathway to identify and analyze these models directly from their parameters, processing full-scale weights is computationally prohibitive. Probing-based methods have emerged as a lightweight alternative, extracting permutation-equivariant representations via learnable probe vectors. However, existing probing methods are limited by a single-view design: they capture first-order structures but fail to encode the rich, higher-order correlation patterns inherent in row--column interactions. To bridge this gap, we introduce MVProbe, a multi-perspective probing framework that synthesizes first-order signals with interaction-aware (Gram-based) views. Our approach is theoretically grounded; we analyze the scaling laws of different probing orders to derive a principled standardization and fusion strategy that ensures balanced contributions from all branches. On the Model Jungle benchmark, MVProbe consistently outperforms the state-of-the-art ProbeX across diverse architectures, including ResNet, SupViT, MAE, and DINO.
BibTeX
@inproceedings{heo2026what,
  author = {Eunwoo Heo and Kyeongkook Seo and Jaejun Yoo},
  title = {{What Linear Probes Miss: Multi-View Probing for Weight-Space Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=vfqrmuVULL},
}
Figure