Learning Representation from Neural Fisher Kernel with Low-rank Approximation

Ruixiang Zhang, Shuangfei Zhai, Etai Littwin, Joshua M. Susskind

TL;DR

This paper introduces the Neural Fisher Kernel (NFK), a principled framework that unifies representation extraction for both supervised and unsupervised models. By leveraging the discovery that these kernels are naturally low-rank, the authors develop an efficient algorithm using Jacobian-vector products to extract compact, high-quality data embeddings that rival the performance of features millions of times larger.

The Problem with Traditional Feature Extraction

In deep learning, "features" are typically equated with the activations of a specific neural network layer. However, this approach has several flaws:

  • Heuristic Selection: Choosing which layer to use is often a manual process of trial and error.
  • Generative Model Ambiguity: In models like GANs or VAEs, it isn't always clear which layer (if any) contains the "best" representation.
  • Information Loss: Layer activations alone may not capture the full power of a learned model. Recent research suggests that gradient-based features—how the model would change if it saw a specific data point—contain rich, untapped information.

The authors seek a principled method for representation extraction that works across different model architectures and learning paradigms (supervised vs. unsupervised).

Methods: The Neural Fisher Kernel

To solve this, the authors turn to the "kernel view" of neural networks, specifically the Fisher Kernel.

Overview of the Neural Fisher Kernel representation extraction pipeline.
  • Neural Fisher Kernel (NFK): They define NFK by reinterpreting various models (GANs, VAEs, and Classifiers) as Energy-Based Models (EBMs). This provides a unified mathematical language to calculate the "Fisher Score"—the gradient of the log-likelihood with respect to the model parameters.
  • The Scalability Challenge: A standard Fisher Vector has the same dimensionality as the number of model parameters. For modern networks, this could be millions or billions of dimensions, making it impossible to use directly.
  • Low-Rank Approximation: The authors empirically demonstrate that NFKs have a low-rank structure (the "Manifold Hypothesis"). They propose an efficient algorithm to compute a truncated Singular Value Decomposition (SVD) of the Fisher matrix.
  • The JVP/VJP Trick: Instead of explicitly calculating a massive $N \times N$ kernel matrix, they use Jacobian-Vector Products (JVP) and Vector-Jacobian Products (VJP). This allows them to find the top $k$ eigenvectors using power iterations at a computational cost similar to standard forward and backward passes.

Key Insights

  • Gradients over Activations: The "Fisher Score" captures how a model's internal knowledge reacts to a specific input. This is more informative than just looking at a static layer activation.
  • The Low-Rank Miracle: Remarkably, for a GAN with 5.9 million parameters, the authors found that just the top 128 dimensions of the NFK could almost perfectly recover the information of the full 5.9M-dimensional Fisher Vector.
  • Unified Theory: By treating a classifier's softmax output as an energy function, they can extract "unsupervised-style" features from a supervised model, and vice versa.

Key Results

The researchers evaluated their method on standard computer vision benchmarks like MNIST, CIFAR-10, and SVHN.

Computation time and scaling efficiency of the proposed low-rank NFK approximation compared to baselines.
  • Efficiency: Their algorithm scales linearly with the number of data points, whereas the baseline (computing the full kernel) fails after $2^{14}$ examples due to memory constraints.
  • Linear Probing: On CIFAR-10, a linear classifier trained on 128-dimensional NFK features from a GAN achieved 89.8% accuracy, matching the performance of the full 5.9M-dimensional feature set and significantly beating standard GAN activations (65.3%).
  • Semi-Supervised Learning: NFK-128d features achieved highly competitive results on CIFAR-10 with very few labels (e.g., 20.68% error with only 500 labels), outperforming several established joint-learning baselines.
  • Knowledge Distillation: Their NFK Distillation (NFKD) method achieved 92.42% accuracy on CIFAR-10, surpassing traditional Knowledge Distillation (91.27%) and other state-of-the-art methods by matching the low-dimensional NFK embedding of a teacher model.

Limitations and Future Work

  • Pre-training Dependency: The method requires a high-quality pre-trained model. If the base model is poor, the extracted NFK features will likely be poor as well.
  • Computational Overhead: While much faster than the baseline, the power iterations still require multiple passes through the network, which is more expensive than a single forward pass used for activation-based features.
  • Diagonal Approximation: To maintain scalability, the authors used a diagonal approximation of the Fisher Information Matrix. Exploring more complex approximations could potentially improve results further.

Summary updated

Abstract

In this paper, we study the representation of neural networks from the view of kernels. We first define the Neural Fisher Kernel (NFK), which is the Fisher Kernel applied to neural networks. We show that NFK can be computed for both supervised and unsupervised learning models, which can serve as a unified tool for representation extraction. Furthermore, we show that practical NFKs exhibit low-rank structures. We then propose an efficient algorithm that computes a low-rank approximation of NFK, which scales to large datasets and networks. We show that the low-rank approximation of NFKs derived from unsupervised generative models and supervised learning models gives rise to high-quality compact representations of data, achieving competitive results on a variety of machine learning tasks.
BibTeX
@inproceedings{zhang2022learning,
  author = {Ruixiang Zhang and Shuangfei Zhai and Etai Littwin and Joshua M. Susskind},
  title = {{Learning Representation from Neural Fisher Kernel with Low-rank Approximation}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=dNXhiYaUQ8},
}
Figure