Learning from Per Sample Training Dynamics

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2154 papers158 conceptsfrozen 2026-10-04

In standard stochastic gradient descent, deep neural networks are remarkably "forgetful." When a model sees a training image in epoch 50, it has no explicit memory of how it classified that same image in epoch 10. The only thing that persists across time is the set of model weights.

However, a growing body of research suggests that the trajectory of how a model treats an individual data point—its "training dynamics"—contains vital information that weights alone cannot capture. By tracking per-sample statistics like prediction history, loss fluctuations, or gradient magnitudes, researchers are building models that are more robust to noise, faster to train, and capable of learning from far less labeled data.

The Power of Retrospection

The core intuition behind this direction is that a model’s past self can serve as a powerful teacher for its current self. In their seminal work on Temporal Ensembling, Samuli Laine and Timo Aila demonstrated that maintaining a running average of predictions for every unlabeled sample allows a model to regularize itself. By forcing the current prediction to be consistent with the historical "consensus," the model becomes less sensitive to the random noise of individual augmentations.

This concept of "retrospection" isn't just for semi-supervised learning. Xiang Deng explored this in Learning with Retrospection, arguing that we should treat every epoch not as a fresh start for the data, but as a continuation of a dialogue. By retaining historical outputs, models can identify "flipping errors"—instances where the model suddenly forgets a correct prediction it had previously mastered—and take corrective action to stabilize learning.

Distinguishing Noise from Difficulty

One of the most difficult tasks in machine learning is telling the difference between a "hard" sample (a blurry image of a cat) and a "noisy" sample (a clear image of a dog accidentally labeled "cat"). Static loss values often fail here, as both types of samples result in high error.

By looking at the history of a sample, however, a pattern emerges. Hwanjun Song and colleagues introduced SELFIE, which identifies "refurbishable" samples by looking for consistent label predictions over time. If a model repeatedly predicts "dog" for a sample labeled "cat," it’s likely the label is wrong. Similarly, the FOCI framework by Jung-Woo Ha and Wonyoung Shin uses historical uncertainty to prioritize samples that are both clean and highly informative.

Accelerating Training via Importance

If we know how a model feels about a sample, we can decide whether it’s even worth looking at. This is the logic behind "dynamic pruning." Abhinav Vishnu proposed sDLr, which tracks individual sample errors to temporarily remove "easy" points that no longer contribute to gradient updates.

More recently, InfoBatch by Ziheng Qin refined this by using importance scores to prune low-loss data while mathematically rescaling the remaining gradients. This ensures the training remains unbiased while slashing the total computational cost by up to 40%.

The Storage Wall

Despite the clear benefits, these methods face a shared, daunting obstacle: the memory bottleneck.

Most deep learning operations scale with the size of the model or the mini-batch. However, tracking training dynamics scales with the size of the entire dataset ($O(N)$). As Raul Castro Fernandez notes in A System for Studying Deep Network Training, recording every parameter change or prediction for every sample across all iterations creates an overwhelming volume of data.

For a dataset like ImageNet with 1.2 million images, storing a simple 1000-class probability vector for every sample across 100 epochs requires terabytes of storage. This "storage wall" is the primary reason why many of these promising techniques struggle to move from research benchmarks to web-scale production environments.

Toward Scalable Dynamics

The frontier of this research is now focused on bypassing this $O(N)$ requirement. New approaches are moving away from full historical buffers toward more efficient estimators. For example, Xidong Wu and Heng Huang developed SLATE, which optimizes complex ranking metrics without storing massive local data states. Similarly, "Hessian-free" unlearning methods, like those proposed by Meng Zhang, attempt to reconstruct the influence of individual data points without maintaining a literal diary of every training step.

By treating the training process not as a sequence of static snapshots, but as a dynamic evolution, these researchers are uncovering a richer way to teach machines—provided they can find a way to remember the past without being buried by it.

Papers in this version

Showing 60 of 154 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 80 papers
  2. Version 1 · 2026-10-04 · 154 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_linear_memory_scaling_per_sample_state_o_e3360663,
  title = {linear memory scaling (per-sample state overhead)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-memory-scaling-per-sample-state-overhead/rdr_e3360663a3264ca2aaac04f19beb157d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_learning_from_per_sample_training_dynami_e3360663_v1,
  title = {Learning from Per Sample Training Dynamics},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learning-from-per-sample-training-dynamics/rdr_e3360663a3264ca2aaac04f19beb157d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}