DIDS: Domain Impact-aware Data Sampling for Large Language Model Training

Weijie Shi, Jipeng Zhang, Yidong Wu, Jingzhi Fang, Shibo Zhang, Yao Zhao, Hao Chen, Ruiyuan Zhang, Yue Cui, Jia Zhu, Sirui Han, Jiajie Xu, Xiaofang Zhou2025 · DOI

Abstract: Weijie Shi, Jipeng Zhang, Yaguang Wu, Jingzhi Fang, Shibo Zhang, Yao Zhao, Hao Chen, Ruiyuan Zhang, Yue Cui, Jia Zhu, Sirui Han, Jiajie Xu, Xiaofang Zhou. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

2025.

TL;DR

DIDS (Domain Impact-aware Data Sampling) is a new framework that treats large language model (LLM) training as a dynamic optimization problem. By grouping data based on how it affects model gradients and using the Fisher Information Matrix to measure downstream impact, DIDS achieves higher performance using only 10% of a dataset than standard training does using the full 100%.

What Problem Does This Paper Address?

Training a state-of-the-art LLM isn't just about having "more data"; it’s about having the right mix of data. Most training sets are multi-domain, containing everything from Python code and mathematical proofs to Wikipedia articles. However, not all domains are equally useful for every downstream task, and their importance changes as the model learns.

The authors identify two major gaps in current "data mixing" strategies:

  1. Intra-domain Inconsistency: We usually group data by its source (e.g., "this is from Reddit"). But two documents from the same source might affect the model in completely different ways. Conversely, a math proof and a snippet of code might have very similar "learning effects" on the model's parameters.
  2. Inaccurate Impact Measurement: Existing methods often use simple "gradient similarity" to see if training data aligns with a target task. This only captures a snapshot in time and doesn't account for how a data update actually shifts the model's predictive behavior or when a domain has reached "diminishing returns."

Methods

DIDS introduces a three-step pipeline to make data sampling smarter and more efficient:

The three-step DIDS pipeline: Data Repartition, Domain Impact Matrix calculation, and Sampling Vector update.
  • Gradient-based Domain Repartition: Instead of relying on metadata labels, DIDS clusters data based on actual training effects. To keep this computationally cheap, they use a tiny "proxy model" (like a 500M parameter Qwen) to generate gradients. They then use Johnson-Lindenstrauss random projection to squash these massive gradient vectors into a manageable size before running K-means clustering.
  • FIM-guided Domain Impact: To measure how much a domain helps a specific task (like GSM8K for math), they use a metric guided by the Fisher Information Matrix (FIM). This quantifies the "distributional shift"—essentially measuring how much a specific data update changes the model's output probabilities on a target task.
  • Dynamic Sampling Strategy: DIDS doesn't just look at what is helping now. It estimates Future Potential by fitting an exponential decay curve to the model's loss history. If a domain’s contribution is saturating, DIDS automatically dials back its sampling ratio to save room for "unsaturated" domains.

Datasets

The researchers primarily used the Tulu-3 post-training dataset (~939,000 samples from 18 sources). For evaluation, they used a suite of nine diverse downstream tasks including BBH, BoolQ, GSM8K, Minerva-MathQA, IFEval, MMLU, PIQA, PubMedQA, and TruthfulQA. They also validated the approach using the OpenHermes-2.5 dataset.

Key Insights and Ideas

  • Gradients are better than labels: Grouping data by how it changes model weights (gradients) is far more effective than grouping by source, ensuring consistent behavior within a "domain."
  • Less is More: DIDS, using only 100k samples, outperformed a model trained on the full 929k samples. This suggests that redundant data in large datasets can act as noise that degrades performance.
  • The Proxy Model Shortcut: A proxy model that is 5-10% the size of the target model provides enough gradient similarity to make excellent sampling decisions, keeping the computational overhead of DIDS to a mere 3.7%.

Key Results

  • Superior Performance: DIDS achieved a 3.4% higher average score across tasks compared to the best baseline (Doge) and significantly outperformed uniform or random sampling.
  • Efficiency Gains: On the MathQA task, DIDS improved the score from 17.8 (strongest baseline) to 20.5.
Performance comparison of DIDS against DGA and Random sampling baselines over training updates.
  • Robustness to Noise: In experiments where the researchers intentionally added "irrelevant" data, DIDS maintained its performance by successfully assigning those domains near-zero sampling weights.

Limitations and Future Work

The authors acknowledge that using a Diagonal FIM Approximation ignores interactions between different parameters. Additionally, their Loss Modeling assumes learning follows an exponential decay curve, which might not hold during "grokking" phases. Finally, they suggest that future iterations should integrate ethical auditing tools to ensure selective sampling doesn't amplify existing biases.

Summary updated

BibTeX
@misc{shi2025dids,
  author = {Weijie Shi and Jipeng Zhang and Yidong Wu and Jingzhi Fang and Shibo Zhang and Yao Zhao and Hao Chen and Ruiyuan Zhang and Yue Cui and Jia Zhu and Sirui Han and Jiajie Xu and Xiaofang Zhou},
  title = {{DIDS: Domain Impact-aware Data Sampling for Large Language Model Training}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.215},
}
Figure