Gradient Based Feature Representations

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 270 papers92 conceptsfrozen 2026-10-04

In traditional machine learning, we represent data by its attributes—the pixels of an image, the words in a sentence, or the activations of a hidden layer. This research direction explores a different perspective: representing a data point by how it affects a model's internal state. Instead of asking "what is this data?", we ask "how would the model's parameters need to change to explain this data?"

By using the partial derivatives (gradients) of a model’s output or loss with respect to its parameters as features, researchers have created a powerful bridge between generative and discriminative modeling, a way to handle variable-length sequences, and a method for interpreting the "black box" of deep networks.

The Model as a Sensor

The core intuition is that a trained model is a sophisticated sensor. When you pass a data point through it, the gradient of the log-likelihood (or loss) with respect to the model's parameters captures the "push" that data point exerts on the model.

As Martin A. Zinkevich and Dale Schuurmans argue in Holographic Feature Representations of Deep Networks, these "holographic" features capture both upstream and downstream information. While a standard activation only knows about the layers below it, a gradient knows about the entire network's logic relative to that specific input. This allows a simple linear model to replicate the complex, non-linear logic of a deep network by simply looking at these gradient-based features.

Bridging Generative and Discriminative Worlds

Historically, this idea emerged as the Fisher Kernel. The problem was that generative models (like Hidden Markov Models) were great at handling structured, variable-length data but lacked the raw predictive power of discriminative classifiers (like Support Vector Machines).

James Henderson and Ivan Titov demonstrated in Data-Defined Kernels for Parse Reranking Derived from Probabilistic Models that you could use a generative model to "define" the features for a discriminative one. By mapping a parse tree to the gradient of its log-likelihood, they transformed a probabilistic model into a high-dimensional feature vector (a Fisher score) that an SVM could easily process.

This mechanism is particularly powerful for sequences. Because the number of parameters in a model is fixed, the gradient vector is always the same size, regardless of whether the input is a three-word sentence or a ten-minute video. Guy Lev and Lior Wolf applied this to Recurrent Neural Networks in RNN Fisher Vectors for Action Recognition and Image Annotation, showing that these gradients capture temporal dynamics more effectively than simple hidden state activations.

The Geometry of Learning

Why are gradients such good descriptors? They represent the tangent space of the parameter manifold. Bálint Daróczy and Andras A Benczur suggest in Expressive power of outer product manifolds on feed-forward neural networks that the sensitivity of a model to an input—how much it "strains" the parameters—is more informative than the raw input itself.

This geometric view has led to breakthroughs in several areas:

  • Active Learning: Alec Hodgkinson uses Fisher Kernels to select unlabeled data that "looks" like a representative validation set in the gradient space, significantly reducing labeling effort.
  • Model Linearization: Wesley J. Maddox and Andrew Gordon Wilson show in Fast Adaptation with Linearized Neural Networks that treating the network's Jacobian (its gradients) as features allows for stable, analytic adaptation to new tasks without the instability of fine-tuning.
  • Data Selection for LLMs: In Diversity Measurement and Subset Selection for Instruction Tuning Datasets, Peiqi Wang and Yoon Kim demonstrate that selecting data based on model weight gradients—rather than semantic text embeddings—leads to better instruction-following performance because it captures the functional relationship between the model and the data.

Security and Interpretability

The sensitivity of gradients also makes them a double-edged sword for privacy. Tianhao Wang shows in White-box Membership Inference Attacks against Diffusion Models that high-dimensional gradient information can reveal whether a specific image was used in a model's training set, even when the scalar loss values are identical for members and non-members.

Conversely, this same sensitivity enables better interpretability. Rajiv Khanna and Been Kim use Fisher kernels to identify which training examples are most responsible for a prediction. By measuring similarity in the gradient space, they find "influential" examples that are intrinsically linked to the model's internal state, providing a more principled explanation than standard feature-based similarity.

Whether used to detect synthetic data, compress massive histopathology images, or speed up LLM training, the shift from activations to gradients treats the model not just as a predictor, but as a rich, high-dimensional coordinate system for the data itself.

Papers in this version

Showing 60 of 70 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 37 papers
  2. Version 1 · 2026-10-04 · 70 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_based_feature_extraction_gradie_eea174d3,
  title = {gradient-based feature extraction (gradient-based feature representation)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-feature-extraction-gradient-based-feature-representation/rdr_eea174d30fe247778de80a30b2fd268f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_based_feature_representations_eea174d3_v1,
  title = {Gradient Based Feature Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-based-feature-representations/rdr_eea174d30fe247778de80a30b2fd268f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}