TL;DR
This paper introduces the Neural Fisher Kernel (NFK), a principled framework that unifies representation extraction for both supervised and unsupervised models. By leveraging the discovery that these kernels are naturally low-rank, the authors develop an efficient algorithm using Jacobian-vector products to extract compact, high-quality data embeddings that rival the performance of features millions of times larger.
The Problem with Traditional Feature Extraction
In deep learning, "features" are typically equated with the activations of a specific neural network layer. However, this approach has several flaws:
- Heuristic Selection: Choosing which layer to use is often a manual process of trial and error.
- Generative Model Ambiguity: In models like GANs or VAEs, it isn't always clear which layer (if any) contains the "best" representation.
- Information Loss: Layer activations alone may not capture the full power of a learned model. Recent research suggests that gradient-based features—how the model would change if it saw a specific data point—contain rich, untapped information.
The authors seek a principled method for representation extraction that works across different model architectures and learning paradigms (supervised vs. unsupervised).
Methods: The Neural Fisher Kernel
To solve this, the authors turn to the "kernel view" of neural networks, specifically the Fisher Kernel.
- Neural Fisher Kernel (NFK): They define NFK by reinterpreting various models (GANs, VAEs, and Classifiers) as Energy-Based Models (EBMs). This provides a unified mathematical language to calculate the "Fisher Score"—the gradient of the log-likelihood with respect to the model parameters.
- The Scalability Challenge: A standard Fisher Vector has the same dimensionality as the number of model parameters. For modern networks, this could be millions or billions of dimensions, making it impossible to use directly.
- Low-Rank Approximation: The authors empirically demonstrate that NFKs have a low-rank structure (the "Manifold Hypothesis"). They propose an efficient algorithm to compute a truncated Singular Value Decomposition (SVD) of the Fisher matrix.
- The JVP/VJP Trick: Instead of explicitly calculating a massive $N \times N$ kernel matrix, they use Jacobian-Vector Products (JVP) and Vector-Jacobian Products (VJP). This allows them to find the top $k$ eigenvectors using power iterations at a computational cost similar to standard forward and backward passes.
Key Insights
- Gradients over Activations: The "Fisher Score" captures how a model's internal knowledge reacts to a specific input. This is more informative than just looking at a static layer activation.
- The Low-Rank Miracle: Remarkably, for a GAN with 5.9 million parameters, the authors found that just the top 128 dimensions of the NFK could almost perfectly recover the information of the full 5.9M-dimensional Fisher Vector.
- Unified Theory: By treating a classifier's softmax output as an energy function, they can extract "unsupervised-style" features from a supervised model, and vice versa.
Key Results
The researchers evaluated their method on standard computer vision benchmarks like MNIST, CIFAR-10, and SVHN.
- Efficiency: Their algorithm scales linearly with the number of data points, whereas the baseline (computing the full kernel) fails after $2^{14}$ examples due to memory constraints.
- Linear Probing: On CIFAR-10, a linear classifier trained on 128-dimensional NFK features from a GAN achieved 89.8% accuracy, matching the performance of the full 5.9M-dimensional feature set and significantly beating standard GAN activations (65.3%).
- Semi-Supervised Learning: NFK-128d features achieved highly competitive results on CIFAR-10 with very few labels (e.g., 20.68% error with only 500 labels), outperforming several established joint-learning baselines.
- Knowledge Distillation: Their NFK Distillation (NFKD) method achieved 92.42% accuracy on CIFAR-10, surpassing traditional Knowledge Distillation (91.27%) and other state-of-the-art methods by matching the low-dimensional NFK embedding of a teacher model.
Limitations and Future Work
- Pre-training Dependency: The method requires a high-quality pre-trained model. If the base model is poor, the extracted NFK features will likely be poor as well.
- Computational Overhead: While much faster than the baseline, the power iterations still require multiple passes through the network, which is more expensive than a single forward pass used for activation-based features.
- Diagonal Approximation: To maintain scalability, the authors used a diagonal approximation of the Fisher Information Matrix. Exploring more complex approximations could potentially improve results further.
Summary updated