Proxy-Based Data Mixture Optimization

This direction is merged into another direction: proxy-based data selection. Its last version stays readable and citable here.

Version 1 of 116 papers25 conceptsfrozen 2026-10-04

When training a Large Language Model (LLM), the "recipe"—the specific ratio of web text, code, academic papers, and books—is often more important than the total volume of data. Historically, researchers determined these ratios through expensive trial-and-error or simple heuristics. However, a growing body of research suggests a more efficient path: the optimal data mixture for a massive model can be discovered using a much smaller "proxy" model.

The Scale-Invariance of Data Importance

The fundamental intuition behind this research direction is that the relative utility of a data domain is largely independent of model size. If a specific mix of Python code and Wikipedia articles helps a 100-million parameter model generalize better, that same mix is likely to be optimal for a 175-billion parameter model.

Sang Michael Xie, Hieu Pham, and their colleagues demonstrated this in DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretraining. They found that proportions optimized on a small proxy model remained effective when transferred to models orders of magnitude larger. This transferability suggests that data synergies—how different datasets complement each other—are a property of the data itself rather than a quirk of model capacity.

From Heuristics to Automated Optimization

Instead of "guessing" ratios, researchers now treat data mixing as an optimization problem. The goal is to find a weight vector for different data domains that minimizes a specific objective, such as the model's "excess loss" or its "generalization error."

Several frameworks have emerged to solve this:

  • Minimax Optimization (DoReMi): DoReMi uses a proxy model to minimize the maximum excess loss across all domains relative to a reference model. This ensures the model doesn't ignore "difficult" but valuable domains.
  • Generalization Estimation (DoGE): In DOGE: Domain Reweighting with Generalization Estimation, Simin Fan, Matteo Pagliardini, and Martin Jaggi propose using a proxy model to estimate which domains best help the model generalize to unseen tasks.
  • Distributionally Robust Optimization (tDRO): For specialized tasks like dense retrieval, Guangyuan Ma and Yongliang Ma introduced tDRO. Their approach measures the "improvement headroom" of each task, automatically focusing training on areas where the model has the most to learn.

Dynamic and Multi-Dimensional Selection

The field is moving beyond static weights toward dynamic systems that adjust data importance in real-time. David Grangier, Pierre Ablin, and Awni Hannun developed a scalable online bilevel optimization framework that selects relevant data on the fly. This allows a model to prioritize different types of data as it matures—for example, focusing on basic language patterns early in training and complex reasoning later.

Furthermore, researchers are looking at more than just "domain" labels. Xinlin Zhuang and the team behind Meta-rater use proxy models to weight 25 different quality dimensions (like toxicity, clarity, and factual density). They found that how you mix these quality metrics is just as critical as the metrics themselves.

Generalizing Across Modalities and Tasks

The success of proxy-based reweighting is not limited to text. In the realm of Vision-Language Models, Wanyun Xie and Volkan Cevher introduced MaD-Mix, which optimizes multi-modal data mixtures in latent space. Similarly, Qiang Cai and Binqiang Pan applied these principles to fine-tuning mobile phone agents in DaMo, showing that even complex multi-task interactions can be balanced using small pilot models.

By offloading the "guessing game" of data curation to efficient proxy models, these researchers have turned a computationally prohibitive bottleneck into a scalable, principled optimization step. The result is faster convergence, better downstream performance, and a deeper understanding of the data that drives model intelligence.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_proxy_based_data_mixture_optimization_510d210c,
  title = {Proxy-Based Data Mixture Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/proxy-based-data-mixture-optimization/rdr_510d210cc72242b5bdd97f58bab2cb30},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_proxy_based_data_mixture_optimization_510d210c_v1,
  title = {Proxy-Based Data Mixture Optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/proxy-based-data-mixture-optimization/rdr_510d210cc72242b5bdd97f58bab2cb30/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}