To train deep learning models efficiently, we must select the highest-quality data. Traditional data selection methods rely on static semantic embeddings to find diverse or representative samples. However, semantic similarity is blind to a model's internal state; it cannot tell us what a model has already mastered or where its decision boundaries remain weak.
To bridge this gap, researchers turn to gradient-based selection. Because a parameter gradient represents the exact directional update a sample provides to a model, it serves as a highly informative proxy for sample uncertainty, diversity, and influence. For instance, in active learning, Jordan T. Ash et al. introduced Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. (BADGE), which scales active learning by embedding unlabeled data into a "hallucinated gradient space." Similarly, Tianyang Wang et al. demonstrated in Boosting Active Learning via Improving Test Performance that selecting samples with the highest gradient norms minimizes the upper bound of test loss.
Yet, a severe paradox emerges. As Mahdi Nikdan et al. point out in Efficient Data Selection at Scale via Influence Distillation, calculating and storing gradients for every sample in a large dataset can become more expensive than simply training on the entire uncurated pool. This computational bottleneck prevents theoretically sound influence-based techniques from scaling to massive foundation models.
The Root of the Overhead: Per-Sample Backpropagation
Why is gradient computation so expensive? Modern hardware accelerators (GPUs and TPUs) are explicitly engineered for batched operations. During standard training, forward and backward passes compute the average gradient across a batch of samples. This allows the hardware to parallelize matrix multiplications efficiently.
Data selection, however, requires per-sample gradients. To evaluate whether a specific instance is informative, redundant, or harmful, we must know its individual impact on the parameters. Computing individual gradients for a pool of $N$ candidate samples requires either running backpropagation with a batch size of 1 for $N$ iterations, or using specialized vectorization techniques that incur massive memory overhead.
This challenge spans across multiple domains:
- Continual Learning: Rahaf Aljundi et al. show in Gradient based sample selection for online continual learning that populating replay buffers using gradient diversity requires expensive forward and backward passes for every candidate sample in a non-stationary stream.
- Instruction Tuning: Peiqi Wang et al. observe in Diversity Measurement and Subset Selection for Instruction Tuning Datasets that evaluating model weight gradients for millions of instruction-tuning data points introduces significant computational overhead, even when paired with dimensionality reduction.
- In-Context Learning: Zhongxiang Sun et al. note in Effective In-Context Example Selection through Data Compression that gradient-based demonstration selection creates a strict trade-off between example quality and inference latency due to the necessity of backpropagation.
Mitigating Overhead via Final-Layer Proxies
The most immediate way to bypass full-network gradient overhead is to restrict gradient computation to the model's final output layer.
The intuition here is simple: the final layer acts as a linear classifier or predictor operating on top of a shared representation space. If two samples produce similar gradients in the final layer, they likely exert similar influences on the model's ultimate predictions.
By computing gradients only with respect to the final-layer parameters, the size of the gradient vector drops from billions of parameters to a fraction of that size. The BADGE algorithm by Jordan T. Ash et al. leverages this by computing final-layer gradients under "hallucinated" labels to capture both uncertainty and diversity. In federated learning, Huancheng Chen proposes Heterogeneity-Guided Client Sampling: Towards Fast and Efficient Non-IID Federated Learning (HiCS-FL), which uses final output layer updates to estimate client data heterogeneity, avoiding the heavy computational and communication burden of analyzing full model gradients.
First-Order and Trajectory Approximations
Another line of work avoids explicit per-sample backpropagation by using mathematical shortcuts like Taylor expansions or tracking loss trajectories over time.
Instead of calculating exact second-order influence functions (which require computing a massive Hessian matrix), the authors of GREATS: Online Selection of High-Quality Data for LLM Training in Every Iteration (Jiachen T. Wang et al.) utilize a greedy Taylor expansion combined with a "ghost inner-product" trick. This allows them to approximate sample informativeness at every single training iteration with almost zero computational overhead.
Alternatively, researchers can look at loss trajectories across training steps rather than explicit parameter updates. In Coresets from Trajectories: Selecting Data via Correlation of Loss Differences, Manish Nagaraj et al. introduce the Correlation of Loss Differences (CLD) metric. By tracking how a sample's loss changes relative to a validation set across training trajectories, CLD identifies highly generalizable coresets without requiring any gradient, Jacobian, or Hessian computations.
Similarly, Ayush Jain et al. propose Train on Validation (ToV): Fast data selection with applications to fine-tuning, which flips the standard influence paradigm. By measuring how training on a small validation set shifts predictions on the wider training pool, they achieve up to a 16x storage reduction and faster runtimes compared to traditional influence functions.
Proxy Models and Gradient Sketching
When dealing with massive foundation models where even a single forward pass is expensive, computing gradients directly on the target model remains impractical. To solve this, researchers employ proxy models and sketching techniques to compress the search space.
For example, in many-shot in-context learning, Jianfei Zhang et al. develop Selecting Demonstrations for Many-Shot In-Context Learning via Gradient Matching. They utilize a smaller, lightweight proxy model to calculate and match the latent gradient trajectory of the training set, transferring the selected demonstrations to larger models like Llama-3-70B.
For large-scale data attribution and unlearning, Ziwen Liu et al. introduce Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning (RASLIK). Recognizing that caching high-dimensional gradients for an entire corpus is storage-prohibitive, RASLIK uses low-dimensional gradient "sketches" to efficiently retrieve and evaluate the influence of training data, drastically expanding the feasibility of data-centric model alignment.
Key Foundational Work
Gradient based sample selection for online continual learning (2019) — contributed 1 concept elements to this direction.
Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds. (2020) — contributed 1 concept elements to this direction.
Lipschitz Generative Adversarial Nets (2019) — contributed 1 concept elements to this direction.