XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML

Ernesto L. Estevanell-Valladares, Suilan Estévez-Velarde, Yoan Gutiérrez, Andrés Montoyo, Ruslan MitkovEMNLP 2025

TL;DR

Fine-tuning large language models (LLMs) is computationally expensive and environmentally taxing. This paper introduces XAutoLM, an AutoML framework that uses meta-learning to efficiently find optimal LLM fine-tuning pipelines by learning from past experiments. XAutoLM significantly reduces search time and computational cost while achieving competitive or superior performance compared to zero-shot methods.

What Problem Does This Paper Address?

Fine-tuning LLMs for specific tasks is crucial for their application, but it's a resource-intensive process. The search space for optimal fine-tuning involves selecting the right language model, the fine-tuning strategy (e.g., full fine-tuning, LoRA), and numerous hyperparameters. Exhaustively searching this space is prohibitively expensive, both in terms of computation and environmental impact. While AutoML has seen success in other domains, a unified approach for LLM fine-tuning that considers both model selection and hyperparameter optimization under tight resource constraints (like limited compute and Green AI goals) has been lacking. Human experts often rely on intuition and past experience to guide this search, a process XAutoLM aims to automate.

Methods

XAutoLM builds upon the AutoGOAL framework, which uses a probabilistic model to explore the search space. The key innovation in XAutoLM is its "warm-start" approach, powered by meta-learning. Instead of starting with a uniform probability distribution over possible pipelines, XAutoLM initializes this distribution based on past experiences.

Here's how it works:

  • Experience Store: A repository stores past pipeline evaluations, annotated with task-specific meta-features (e.g., dataset size, class imbalance) and system meta-features (e.g., available RAM, GPU VRAM).
  • Meta-Learning Prior: When a new task arrives, XAutoLM retrieves relevant past experiences. It then computes task and system meta-features for the current task and uses this information to adjust the search space. This adjustment is done by probabilistically shifting the search towards configurations that were successful in similar past tasks and away from those that led to failures or dead ends.
  • Adaptive Pull/Push: The framework uses "positive" experiences (successful runs) to pull the search towards promising regions and "negative" experiences (failed runs) to push it away from costly dead ends. The strength of these pulls and pushes is adaptively determined based on the similarity between the current task and past experiences.
  • Utility Functions: XAutoLM employs different utility functions (Weighted Sum, Linear Front, Logarithmic Front) to prioritize past experiences based on their performance and efficiency metrics.

This meta-learning approach allows XAutoLM to learn from both successes and failures, creating an "experience-aware prior" that guides the AutoML search more efficiently.

Datasets

The paper evaluates XAutoLM on the following datasets:

  • Text Classification:
    • LIAR (6 classes, ~13k claims)
    • SST-2 (binary, ~68k examples)
    • MELD (7 emotions, ~14k utterances)
    • AG News (4 classes, ~128k news articles)
  • Question Answering:
    • SQuAD 1.1
    • DROP

These datasets were chosen to represent a range of challenges, including varying sample sizes, label entropy, and document lengths. Experiences were exchanged only within task families (text classification to text classification, QA to QA) to ensure transfer learning from relevant meta-knowledge.

Key Insights and Ideas

  • Meta-learning for Efficiency: The core idea that learning from past experiments (meta-learning) can dramatically accelerate the search for optimal LLM fine-tuning pipelines is a key takeaway. It's not just about finding a good pipeline, but finding it faster and with less computation.
  • Learning from Failures is Crucial: XAutoLM's ability to learn from "negative traces" (failed runs) is highlighted as essential. This prevents the optimizer from repeatedly exploring known problematic configurations, saving significant resources.
  • Task Similarity Matters: The effectiveness of the warm-start prior is strongly linked to the similarity of meta-features between tasks. When tasks are similar (e.g., LIAR and MELD), the prior leads to rapid convergence. Even with less similar tasks, it still provides benefits.
  • Green AI Alignment: The framework directly addresses the growing concern for "Green AI" by reducing the computational overhead and carbon footprint associated with LLM fine-tuning.
  • Beyond Zero-Shot: XAutoLM consistently outperforms zero-shot optimization, demonstrating that a guided search informed by past experience is superior to random exploration or simple baselines.

Key Results

XAutoLM demonstrated significant improvements across multiple benchmarks:

  • Performance: Surpassed zero-shot optimizer's peak F1 score on five out of six tasks.
  • Efficiency: Cut mean pipeline evaluation time by up to 4.5x.
  • Error Reduction: Reduced search error ratios by up to sevenfold.
  • Pareto Front Improvement: Uncovered up to 50% more pipelines above the zero-shot Pareto front, indicating better performance-time trade-offs.
  • Transfer Learning: Showed effective transfer of knowledge across related tasks, leading to faster convergence and better results.
  • Robustness: Even with limited relevant experiences, XAutoLM's adaptive approach provided benefits.

Limitations and Future Work

  • Scaling to Bigger LLMs: The current implementation is primarily evaluated on single GPUs, limiting the size of LLMs that can be tested. Future work could involve extending XAutoLM to support multi-GPU setups and parameter-efficient evaluation backends.
  • Multimodality: The current experience store and benchmarks are text-only. Future work should investigate the transferability of the warm-start prior to dialogue, speech, or multimodal pipelines.
  • Statistical Significance: While promising results were observed, statistical significance testing for multi-objective fronts requires more extensive runs, which is left for future work.
  • Energy Measurement: The paper relies on empirical links between execution time and power draw from prior work. Future releases will directly log wattage and CO2 estimates.

Summary updated

BibTeX
@inproceedings{estevanellvalladares2025xautolm,
  author = {Ernesto L. Estevanell-Valladares and Suilan Estévez-Velarde and Yoan Gutiérrez and Andrés Montoyo and Ruslan Mitkov},
  title = {{XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure