Abstract
In this paper, we study the representation of neural networks from the view of kernels. We first define the Neural Fisher Kernel (NFK), which is the Fisher Kernel applied to neural networks. We show that NFK can be computed for both supervised and unsupervised learning models, which can serve as a unified tool for representation extraction. Furthermore, we show that practical NFKs exhibit low-rank structures. We then propose an efficient algorithm that computes a low rank approximation of NFK, which scales to large datasets and networks. We show that the low-rank approximation of NFKs derived from unsupervised generative models and supervised learning models gives rise to high-quality compact representations of data, achieving competitive results on a variety of machine learning tasks.
Introduction
In deep learning, data representations are typically extracted as activations from specific layers of pre-trained models. However, this heuristic approach often fails to capture the full breadth of a model's knowledge. Recent research indicates that gradient-based features—specifically Fisher Vectors, which describe how a model would adapt to a given data point—offer a much richer information source than simple activations.
The primary obstacle to using Fisher Vectors is their dimensionality, which equals the number of model parameters (often in the millions). This makes them computationally prohibitive for large-scale applications. This paper introduces the Neural Fisher Kernel (NFK), a framework that leverages efficient low-rank approximations to extract high-quality, compact embeddings from any pre-trained neural network, whether supervised or unsupervised.
The Neural Fisher Kernel Framework
The authors unify supervised classifiers and unsupervised generative models (such as GANs and VAEs) by treating them as Energy-Based Models (EBMs). This unification allows for the derivation of a consistent Fisher Kernel across diverse architectures. The NFK measures the similarity between data points based on the model's parameter response to those points.
To address the dimensionality challenge, the paper proposes an algorithm utilizing Power Iteration combined with Jacobian-Vector Products (JVP) and Vector-Jacobian Products (VJP). This approach avoids the explicit construction of the massive $N \times N$ kernel matrix, enabling the method to scale linearly with the number of data examples while maintaining constant memory usage.
Low-Rank Structure and Efficiency
A key insight of this work is the "Low-Rank Miracle": practical NFKs are naturally low-rank. In experiments with a DCGAN trained on MNIST, the authors found that the first 100 principal components of the Fisher Vector matrix explained 99.5% of the total variance.
This property justifies the use of low-rank approximations, as it allows for massive compression—throwing away over 99.9% of dimensions—while retaining nearly all semantic information. This efficiency is critical for applying gradient-based features to modern "Big Data" contexts and deep architectures like Wide ResNets.
Experimental Performance
The NFK representations were evaluated across several benchmarks, including MNIST, CIFAR-10, and SVHN:
- Massive Compression: On CIFAR-10, a 128-dimensional NFK embedding matched the performance of the full 5.9 million-dimensional Fisher Vector.
- Semi-Supervised Learning: With limited labels (e.g., 500 for CIFAR-10), NFK-based representations outperformed classic methods like Mixup and VAT, remaining competitive with state-of-the-art techniques like MixMatch.
- Knowledge Distillation: In teacher-student setups, using NFK embeddings as distillation targets (NFKD) achieved higher accuracy (92.42%) than standard logit-based distillation (91.27%).
- GAN Inversion: 100-dimensional NFK embeddings proved semantically complete enough to reconstruct original images from a GAN more effectively than standard PCA embeddings.
Conclusion
The Neural Fisher Kernel provides a principled and unified tool for representation extraction. By exploiting the low-rank nature of these kernels, the proposed method overcomes the computational barriers of gradient-based features. While the current implementation relies on high-quality pre-trained models and diagonal approximations of the Fisher Information Matrix, it sets a strong foundation for future exploration in other domains like Natural Language Processing.
Summary updated