Predicting the performance of new Large Language Models (LLMs) typically involves expensive and time-consuming benchmarking. This paper introduces Lineage-Regularized Matrix Factorization (LRMF), a novel method that leverages a model’s "family tree"—its fine-tuning or merging history—to accurately forecast performance. By incorporating these ancestral ties, researchers can predict a new model's scores with significantly higher correlation than standard methods, even before the model is fully evaluated.
The Challenge of LLM Performance Prediction
Building LLMs is resource-intensive. Developers often face "what if" scenarios regarding fine-tuning on specific datasets or merging various checkpoints. Traditionally, determining the outcome requires substantial compute for both model creation and exhaustive benchmarking.
While "scaling laws" provide coarse predictions based on parameter size or training tokens, they fail to capture the nuances of fine-tuning or model merging. Standard recommendation algorithms, such as those used in collaborative filtering, suffer from a "cold-start" problem: they cannot predict the performance of a new model without historical data. This research bridges this gap by treating model development as a genealogical process.
Methodology: Lineage-Regularized Matrix Factorization
The authors propose Lineage-Regularized Matrix Factorization (LRMF), which operates through several key steps:
- Matrix Factorization: Similar to movie recommendation systems, a matrix is constructed where rows represent models and columns represent specific benchmark tasks.
- Lineage Graph Construction: Metadata from Hugging Face is used to build a graph of LLM relationships. Edges are drawn between models if one is a fine-tuned version or a merge of others.
- Graph Laplacian Regularization: This mathematical constraint forces the latent embeddings of parent and child models to remain close, ensuring that performance characteristics are inherited.
- Multi-hop Ancestry Propagation: Unlike simple averaging, LRMF propagates performance signals across multiple generations, allowing a "grandparent" model's strengths to inform a "grandchild's" predicted performance.
A simpler baseline, Model Lineage Averaging (MLA), was also tested, which estimates a child's performance by simply averaging its parents' scores.
Experimental Setup
The study utilized a massive dataset from the Hugging Face Open LLM Leaderboard v2, encompassing:
- 2,934 Models: Publicly available LLMs.
- 21,000+ Instances: Individual test items across benchmarks.
- 6 Major Benchmarks: Including BBH (reasoning), GPQA (science), IFEval (instruction following), MATH, MMLU-Pro, and MuSR.
- Lineage Data: Extracted from model cards and configuration files (e.g.,
base_modeltags).
Key Results and Insights
The researchers found that lineage is a high-fidelity signal for performance. Explicitly knowing a model's parents is a much stronger predictor than coarse metadata like parameter count. Furthermore, inheritance is task-dependent; specialized knowledge (GPQA) and complex reasoning (MATH) are highly heritable, whereas instruction-following (IFEval) tends to change significantly during fine-tuning.
Key quantitative findings include:
- Improved Correlation: LRMF improved the Pearson correlation coefficient by 0.15 to 0.30 over standard Neural Collaborative Filtering (NCF).
- Effective Model Routing: In "routing" experiments, LRMF-based strategies outperformed the "Best Model" baseline, correctly identifying which specific model variant is best for a given task.
- Data Efficiency: LRMF achieves high accuracy with only 50–100 evaluated instances per model, representing a 5x to 10x reduction in required evaluation data.
Limitations and Future Directions
While powerful, the method has limitations. It relies on a "family tree"; models trained from scratch without lineage default to standard prediction methods. Additionally, the current model treats all lineage links equally, whereas different types of connections (e.g., LoRA vs. full fine-tune) might warrant different weights. Finally, the system's accuracy is dependent on the quality of metadata provided by developers on platforms like Hugging Face. Future work will focus on weighting lineage connections and improving robustness to noisy metadata.
Summary updated