Influence-Guided Data Selection with Adaptive Reweighting for Efficient LLM Distillation
Lists
Only you see this
Abstract
Inefficient data selection in LLM distillation—characterized by poor sample utility, redundant information, and misalignment with student model needs—often leads to slow convergence and suboptimal performance. Traditional methods (e.g., random sampling, perplexity filtering) overlook the causal impact of individual samples on distillation outcomes, limiting their ability to prioritize high-value instances. We present InfCurator, an automated data curation framework that leverages influence function gradients to quantify a sample’s true contribution to the student model’s learning, combined with an adaptive reweighting mechanism to optimize training dynamics. InfCurator features three synergistic modules: (1) an Influence Gradient Estimator that computes the marginal effect of each sample on distillation loss via first-order Taylor approximation, identifying samples that maximize performance gain while minimizing overhead; (2) an Adaptive Reweighting Scheduler that adjusts sample weights in real time based on the student’s validation loss—amplifying weights for high-influence samples and damping those with negative marginal contributions; (3) a Redundancy Mitigator that uses contrastive divergence to eliminate semantically similar samples, ensuring the training set remains diverse and informative. For computational efficiency, we incorporate a Pretrained Value Scorer (trained on a small subset of influence values) to rapidly rank samples, cutting selection overhead by 57%. Experiments across GPT-2, Gemma, and Qwen2.5 demonstrate that InfCurator achieves 29% faster convergence and a 9.7% boost in downstream performance using only 41% of the original data, outperforming baselines like DPP-based selection and perplexity filtering. This work provides a theoretically grounded, influence-driven approach to data curation, advancing efficient and high-fidelity LLM distillation.
BibTeX
@misc{li2026influence,
author = {Lujun Li},
title = {{Influence-Guided Data Selection with Adaptive Reweighting for Efficient LLM Distillation}},
year = {2026},
url = {https://openreview.net/forum?id=dDFVEEBsfa},
howpublished = {OpenReview},
}Loading related content...