Approximating Model Retraining with Influence Functions

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2188 papers210 conceptsfrozen 2026-10-04

In modern machine learning, we often need to know how a model would change if we altered its training data. If a user invokes their "right to be forgotten" under GDPR, we must remove their data. If we want to perform cross-validation, we must train the model multiple times on different subsets. If a model makes a mistake, we want to know which specific training examples caused it.

The "brute force" solution is to retrain the model from scratch for every change. However, for large-scale neural networks, this is computationally prohibitive. A growing body of research focuses on Influence Functions, a technique from robust statistics that allows us to "virtually" retrain a model in seconds by approximating the effect of data perturbations.

The Intuition: Navigating the Loss Landscape

To understand how influence functions work, imagine the training process as finding the lowest point in a vast, hilly landscape (the loss surface). The final model parameters $\theta$ sit at the bottom of a valley.

When we remove a training point, we are slightly changing the shape of that landscape. The old valley might shift a few inches to the left. Influence functions ask: Without re-running the entire optimization, can we predict where the new valley will be?

As Yong Liu and Shizhong Liao demonstrate in Efficient Approximation of Cross-Validation for Kernel Methods using Bouligand Influence Function, we can treat the removal of data as a mathematical perturbation. By calculating the gradient (the direction of the slope) and the Hessian (the curvature of the valley), we can estimate the shift in parameters using a Taylor expansion.

The Mechanism: Gradients and Curvature

The core formula for influence involves two components:

  1. The Gradient: How much does this specific data point "pull" the model parameters?
  2. The Inverse Hessian: How "stiff" or "flexible" is the model at its current state?

By multiplying the gradient of a specific point by the inverse of the Hessian matrix, we calculate a "nudge" that moves the parameters toward the state they would have reached if that point were never there.

Researchers like Lijie Hu and Chenyang Ren apply this in Editable Concept Bottleneck Models to surgically update models—fixing mislabeled concepts or deleting specific data points in minutes rather than hours. Similarly, Dharmesh Tailor and Christos Louizos use a Gauss-Newton approximation of the Hessian in Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence to provide rigorous uncertainty estimates without the cost of leave-one-out retraining.

Machine Unlearning: The Right to be Forgotten

One of the most critical applications of this research is Machine Unlearning. As surveyed by Heng Xu and Tianqing Zhu in Machine Unlearning: A Survey, influence functions allow a model to "forget" a user's data by applying a calculated offset to the model weights.

This isn't just about random data removal. Ling Han and Hao Huang show in Unlearning Information Bottleneck that influence functions can be used to remove systematic biases or patterns. By estimating the influence of biased samples, the model can be recalibrated to a "fairer" state without starting from scratch.

Data Attribution and Debugging

Influence functions also serve as a powerful "microscope" for model interpretability. They tell us which training examples are most responsible for a specific prediction.

  • Gradient Rollback: Mathias Niepert and Carolin Lawrence proposed Gradient Rollback, which tracks the history of parameter updates. To see the effect of removing a point, they simply "subtract" its accumulated gradients from the final model.
  • Data Poisoning: Attackers can use influence functions to find the most "fragile" points in a dataset. In Practical Data Poisoning Attack against Next-Item Recommendation, Hengtong Zhang and Bolin Ding use these approximations to efficiently generate adversarial behavior that manipulates recommendation rankings.

Challenges in High Dimensions

While theoretically elegant, influence functions face hurdles in modern deep learning. Standard Taylor expansions can fail when the number of parameters is massive. Andrea Schioppa and Ivan Titov point out in Theoretical and Practical Perspectives on what Influence Functions Do that "parameter divergence" can occur, where the influence of a point fades or becomes erratic over long training trajectories.

To combat this, Haolin Zou and Arnab Auddy introduced Newfluence, which uses a "single-Newton-step" correction. Instead of a simple linear approximation, it calculates one optimization step toward the hypothetical retrained state, restoring accuracy in high-dimensional settings.

Summary

The research direction of influence-based approximation transforms the static "black box" of a trained model into a dynamic, editable structure. By leveraging the local geometry of the loss surface—specifically gradients and curvature—researchers can simulate the complex process of retraining with a fraction of the computational cost. This enables a future where models are not just trained once and deployed, but are continuously refined, audited, and "unlearned" in real-time.

Papers in this version

Showing 60 of 188 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 130 papers
  2. Version 1 · 2026-10-04 · 188 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_influence_based_parameter_estimation_86ff9969,
  title = {influence-based parameter estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/influence-based-parameter-estimation/rdr_86ff996968bf491593ae51b17e8013f8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_approximating_model_retraining_with_infl_86ff9969_v1,
  title = {Approximating Model Retraining with Influence Functions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/approximating-model-retraining-with-influence-functions/rdr_86ff996968bf491593ae51b17e8013f8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}